git://git.whamcloud.com - fs/lustre-release.git/log

LU-15846 tests: don't use comma-separated debug flags

To avoid test interop issues between b_es6_0 clients and b_es5_2
servers, don't use comma-separated debug flags in sanity-quota.sh
quota_init() and quota_fini().

Lustre-change: https://review.whamcloud.com/47308
Lustre-commit: TBD (from 12282314e2567973b1a7cb4d6806e987bf1a6755)

Test-Parameters: trivial testlist=sanity-quota.sh serverversion=2.12.6-ddn66
Fixes: 6b6fde1026 ("LU-13055 libcfs: allow comma-separated masks")
Fixes: 78be823f33 ("LU-15218 quota: delete unused quota ID")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifca39054d14292bca8bcff9b8e03ae58fd5cc3a8
Reviewed-on: https://review.whamcloud.com/47307
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15285 mdt: fix same-dir racing rename deadlock

With LU-12125 lifting the BFL for same directory rename,
a deadlock possibility opens up since we lock source and target
of rename in the source-target order, if there are two renames
racing to rename arguments in reverse order:
mv a b &
mv b a

a lock inversion happens and a deadlock has been observed.

To avert this - instill additional order requirement:
lower PDO hash value is to be locked ahead of the higher one.

Lustre-change: https://review.whamcloud.com/45676
Lustre-commit: 82ec537d8b4cc9261828f4efe6b03d8d33f38432

Fixes: d76cc65d5d68 ("LU-12125 mds: allow parallel regular file rename")
Fixes: b50bb830f92e ("LU-3538 dne: Commit-on-Sharing for DNE")
Fixes: 9f1711f3d7d1 ("LU-12081 mdt: rename shouldn't PDO lock if parent is remote")
Change-Id: I88dd3aebb394ea40e97e6029d6dcc161116f982e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47278
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4973 lipe: remove changelog user registration from config

In stratagem-hp-config.sh, remove the changelog user registration
code. If no chnagelog user is configured then lamigo will register a
new named changelog user with the required mask.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I457f4539451e93bf7980fc7fb807f67e67abf247
Reviewed-on: https://review.whamcloud.com/47045
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

EX-5176 utils: fix package requirements for json-c

SLES15.3 uses "libjson-c3" for json-c RPM vs. "json-c" for RHEL.
Ubuntu 20.04 uses "libjson-c4" and 22.04 uses "libjson-c5", but
the right one is depended on by "libjson-c-dev".

Test-Parameters: trivial testlist=sanity clientdistro=sles15sp3
Test-Parameters: testlist=sanity clientdistro=ubuntu2004
Fixes: fbfd2d075534 ("EX-5176 pcc: use JSON string for trusted.pin xattr")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibb01cfb7dadda83572be68961cf98450536980cf
Reviewed-on: https://review.whamcloud.com/47267
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5235 lipe: Fix lipe-server build

To compile against liblustreapi, we now also need json-c.

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I5e5323c556d1d3bdd4fd6d7f9bc59e33baf59310
Reviewed-on: https://review.whamcloud.com/47263
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15025 quota: stale edquot after clearing limits

When hard and soft limit set to 0, lqe enforced flag is also
set to false. As qmt_adjust_qunit does not handle not enforced
lqes, edquot set to the pool continues to be true and a user
gets -EDQUOT even if all pool limits are cleared. This was ok
for global pool lqe as since it turned off, zero limits are
sent to OSTs causing OSTs to release all granted space and
avoid EDQUOT. Fix this for PQ - set edquot and qunit to zero,
since appropriate lqe becomes "not enforced".

Lustre-change: https://review.whamcloud.com/45000
Lustre-commit: 6ed5d6ed3bd0a705c6ae8c37a0b7e8906ebc6079

HPE-bug-id: LUS-10146
Change-Id: I1e08929bae7e1b37b1e8cbbc44859a786b5fb090
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/46657
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15787 sec: document enc-unaware clients on enc files

Document the behavior of encryption-unaware clients when they access
encrypted files.

Lustre-change: https://review.whamcloud.com/47182
Lustre-commit: 751a8114ef3afe9abe7692b3974b070db6a705a2

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0354e3051e10aa0542baeb8e34c6201d47e65710
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47228
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15803 sec: correctly handle page lock in ll_io_zero_page

In ll_io_zero_page(), we need to make sure we have locked the page,
and it is up-to-date, before zeroing. So modify ll_io_read_page()
behavior to not disown the clpage for our use case. It avoids being
exposed to concurrent modifications.

Lustre-change: https://review.whamcloud.com/47170
Lustre-commit: edcd05e5ac035dd1daf263105df33d7cfc6dbf6e

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I58e4cf80374a798c9c4302364cf2fb39da9033bb
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47229
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14802 nodemap: return proper error code

In nodemap_add_range_helper() it was always returning -ENOMEM when
there was an error inserting a new range into the existing nodemap.

    nodemap_add_range_helper()) cannot insert nodemap range: rc = -17
    mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command: rc = -12

This was confusing because the error returned by range_insert() was
typically -EEXIST (i.e. the entry being inserted already was in the
nodemap).  Do not print an error to the console in this common case.

Return the actual error to the caller so that this is more clear
to the end user.  Have l_ioctl() always set errno on error, in
addition to returning the error, since many callers depend on this.

Lustre-change: https://review.whamcloud.com/45626
Lustre-commit: 7d6cddff24de2f79e539eae462554dc21e674511

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2c80a11dfdf9e6e1c9a8235b8f74f5bcea68c08e
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-on: https://review.whamcloud.com/47247
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15787 sec: block enc unaware clients on enc files

Prevent encryption unaware clients from manipulating encrypted files
and directories. Those can be old clients, or clients built without
encryption support (intentionally or because they run on an old
kernel).
In the mdt layer, check that clients have the OBD_CONNECT2_ENCRYPT
connection flag, and if not, block access if they try to manipulate
a file or directory that has the LUSTRE_ENCRYPT_FL flag.
The forbidden operations from encryption unaware clients are:
- open
- create
- link
- rename
- migrate
Improve sanity-sec test_54 to test this use case.

Lustre-change: https://review.whamcloud.com/47156
Lustre-commit: a31db2ec062ccc995527d37f0330edbca9d486a9

Test-Parameters: testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 serverdistro=el7.9
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ief0639e49c0a8e1a1a0cb19cb13c006edfdff6c4
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47227
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn42

New tag 2.14.0-ddn42

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifcaf23d84abf5802ab305e242d6ec3b3936d837e

LU-14790 tests: Check NI status when link is downed

Add test to check that NI status is set to down when the
ni_fatal_error_on flag is set (i.e. when a link is down).

Lustre-change: https://review.whamcloud.com/44073
Lustre-commit: 412970c586ea189d442620316c5ed6f1fa43cc0e

HPE-bug-id: LUS-10167
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: If98a899b0ee8dd9637c08774109668ad06244c60
Reviewed-on: https://review.whamcloud.com/46846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14790 lnet: Reflect ni_fatal in NI status

If the ni_fatal_error_on flag is set on an NI then that NI should be
considered down.

Lustre-change: https://review.whamcloud.com/44072
Lustre-commit: d77e95cc6d4e947be60757a861c656d8b30d2972

HPE-bug-id: LUS-10167
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I201bda7e06da1fb1cc23db70ce0cfa3118635d0f
Reviewed-on: https://review.whamcloud.com/46845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5176 pcc: use JSON string for trusted.pin xattr

Change the format of trusted.pin xattr to JSON.
For example:
trusted.pin=[{"hsm":2}]
trusted.pin=[{"hsm":2},{"hsm":3},{"pool":"ddn_ssd"}]

Change-Id: I38aeb4ad960cf8324e77ac467f5e5cd742f9ec86
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47161
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15593 tests: fix version checks for backported patches

Check for 2.14.0.x versions instead of 2.14.5x versions:
   conf-sanity.sh: test_123ag
   sanity.sh: test_27S, test_64i, test_160q, test_413e
   sanity-flr.sh: test_44e
   sanity-sec.sh: test_60

Test-Parameters: trivial testlist=conf-sanity env=ONLY=123
Test-Parameters: testlist=sanity-flr,sanity-sec
Fixes: 32f0bbe2646a ("LU-15176 sec: allow subdir mount of encrypted dir")
Fixes: d053ce30ca2f ("LU-15593 mdt: Add option to disable use of SOM")
Fixes: a35ae4ec46ec ("LU-15142 lctl: fixes for set_param -P and llog_print")
Fixes: d35aa94e79de ("LU-14448 lod: verify LOV before set/inherit")
Fixes: 276182ae9b95 ("LU-13055 changelog: use default mask if server has no mask")
Fixes: 930d4fe4952f ("LU-9704 grant: ignore grant info on read resend")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibd0e01ff5ac3b4a810b3eb5b8052e8a727ca416d
Reviewed-on: https://review.whamcloud.com/47206
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14889 lproc: Add server checksum_type

This patch adds 'checksum_type' lproc entries under server:
1. obdfilter.$FSNAME-OST${count}.checksum_type
2. mdt.$FSNAME-MDT${count}.checksum_type

Test-case: sanity/77o added

Lustre-change: https://review.whamcloud.com/44755
Lustre-commit: c18d5d892b62ccee3cc9ec20a2278f17bb2d2f2c

Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I12a26f5c8c8f2f93d57f377626b1753fc13ffbb3
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/47142
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5199 sec: reinstate null encryption for file names

Reinstate null encryption for file names by adding a new llite
parameter named 'enable_filename_encryption', set to 0 by default.
When this parameter is 0, new empty directories configured as
encrypted ignore the filenames_encryption_mode and use
LLCRYPT_MODE_NULL instead, which is a no-op. This LLCRYPT_MODE_NULL
mode is inherited for all subdirectories and files.
When this parameter is 1, new empty directories configured as
encrypted use the normal encryption mode.

To set this parameter globally for all clients, do on the MGS:
mgs# lctl set_param -P llite.*.enable_filename_encryption=0

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9d726ba26cc91a51690d59a81efe3eb98ee2995c
Reviewed-on: https://review.whamcloud.com/47184
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15810 sec: fix enc dir migration

Now that the encryption context is stored in an xattr named
"encryption.c" instead of "security.c", we need to fetch this xattr
explicitly in case of encrypted directory migration. Indeed, there is
no xattr handler in ldiskfs for this "encryption." xattr type, so it
is not returned when listing all xattrs to migrate.

Lustre-change: https://review.whamcloud.com/47201
Lustre-commit: TBD (540f9208a9f6edcf6fdd1711e86fc182dfb92298)

Fixes: 4231fab66e ("LU-13717 sec: make client encryption compatible with ext4")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I628f9b253e86343db0b71f6a5b1ad2c5728ca38d
Reviewed-on: https://review.whamcloud.com/47207
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12373 utils: fix llapi_pcc_del_internal() flags

The pch_flags value used by llapi_pcc_del_internal() was declared
as a "bool" but is used to pass a __u32 flags value from the caller.
This coincidentally works in the case where "flags = 1" (which is
the only value currently in use), but would break in confusing ways
in the future.

Change pch_flags to be "enum lu_pcc_cleanup_flags" to make it clear
what possible flag values are being used in this interface.

Fixes: 871da4898405 ("LU-12373 pcc: uncache the pcc copies when remove a PCC backend")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I41a008856af396376303fb156e2a92bce53ebbe5
Reviewed-on: https://review.whamcloud.com/47123
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>

LU-15761 obdclass: fix locking in llog_cat_refresh()

the patch fixes two problems:
1) pairing up_write() should be used with cathandle
2) llog_read_header() manipulates loghandle's internal
   structures (header, last_idx, etc) which are supposed
   to stay consistent from another user's point of view
   (like llog_add_rec())

Lustre-change: https://review.whamcloud.com/47185
Lustre-commit: 69cdcbe3cf732124b3b5be1a1f235aa1b78f6c85

Fixes: 71f409c9b31b ("LU-11418 llog: refresh remote llog upon -ESTALE")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ib86e10a925b541d02c22d74e6ddbc4368345ac11
Reviewed-on: https://review.whamcloud.com/47190
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15815 llite: disable fast_read and workaround

Revert the fast_read stale data workaround from LU-14541 and disable
fast_read by default. The workaround causes applications to receive
spurious SIGBUGs when reclaim is concurrent with mmap page fault
handlers. We disable fast read to avoid the stale data issue entirely.

This reverts commit f2a16793fa4316fc9ccdc46bcfe54f6b8d1e442b and
re-exposes us to the consistency issues described in LU-14541.

Lustre-change: https://review.whamcloud.com/47204

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I716b1fd6ab22242b9267b8883f0371a360aaecef
Reviewed-on: https://review.whamcloud.com/47205
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn41 and kernel ddn17

Increase kernel build version to -ddn17.

New tag 2.14.0-ddn41

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3ebbe53c8a8d3c02d2831b8ac6d3068c0b3ebbe5

RM-620 build: New tag 2.14.0-ddn40

New tag 2.14.0-ddn40

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3a069f3c8a8d3c02d2831b8ac6d3068c0be50a64

DDN-1956 tests: exercise laudit/laudit-report

Add sanity-laudit.sh to exercise laudit and laudit-report command
line tools.

Test-Parameters: trivial
Test-Parameters: mdtcount=2 testlist=sanity-laudit serverextra_install_params="--packages lipe-client" facet=mds1
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I97fefecab6442d9031de6a7fcd86d07a517386c2
Reviewed-on: https://review.whamcloud.com/46907
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14156 utils: mirror split to check for last in-sync early

currently this check to prevent last in-sync component is done
once the file is open with O_RDWR which interrupts on-going
resync/extend process. instead we can do this check early once
the layout is fetched (after the first open with O_RDONLY).

Lustre-change: https://review.whamcloud.com/40782
Lustre-commit: TBD (from 9996c22e346c219df5b165eb5ceab09873303bc2)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iee08d23008b44d2a7b2127358116a95ace40b7dd
Reviewed-on: https://review.whamcloud.com/47058
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15733 kernel: kernel update RHEL7.9 [3.10.0-1160.62.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.62.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I063ebf0434dceb023847f49f37d6217f5c719b58
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47063
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15536 utils: add lfs somsync utility

This patch adds lfs somsync utility to synchronize
SOM xattr(s) for given FILE(s) or FID(s).

lfs somsync FILE ...
lfs somsync --by-fid MOUNT FID ...

Test-Parameters: trivial env=ONLY=807 testlist=sanity

Change-Id: Ie9ee39625d56ec026c89dcc0f27025904ca354e3
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46942
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14110 obdclass: Protect cl_env_percpu[]

cl_env_percpu is not protected against multi client mounts on the
same node: "keys_fill" could be called with the same cl_env_percpu
context by several mount processes (race on lu_context.lc_value).

This patch add a mutex for cl_env_percpu to proctect contexts
"refill".

Lustre-change: https://review.whamcloud.com/40565
Lustre-commit: 881551fbb7335694b89a877072bcda0aeaf8705c

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: Icfd6f3715899fa4ac5279e932f462e7cf29d98bd
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47073
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15145 hsm: unlock the restore layout lock for a cancel

The HSM restore EX layout lock is not unlock by a HSM cancel action
or by "hsm_control=purge" parameter.

This patch call cdt_restore_handle_del() in mdt_cancel_all_cb() and
mdt_agent_record_update_cb() for restore action (when updating action
status to ARS_CANCELED).
The test "sanity-hsm test_103a" checks the "purge actions" with
blocking restore.

Lustre-change: https://review.whamcloud.com/45341
Lustre-commit: 6d4019281b392bcb6993d1cfca3d47d7fa5f7c56

Test-Parameters: testlist=sanity-hsm,sanity-hsm
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Id891e06aacd2a2c5950048a2d2a5d1398eedfdd7
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46932
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14958 kernel: use rhashtable for revoke records in jbd2

resizable hashtable should improve journal replay time when
the latter has got million of revoke records

before:
1048576 records - 95 seconds
2097152 records - 580 seconds

after:
1048576 records - 2 seconds
2097152 records - 3 seconds
4194304 records - 7 seconds

Lustre-change: https://review.whamcloud.com/45122
Lustre-commit: TBD (from cc2996585c489632fc8fb02ebf61d755a339361a)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9a9e3801223fa9e36cbf6d2ef5ddbad5dff3e19d
Reviewed-on: https://review.whamcloud.com/46371
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 build: remove parallel configure dir

This patch removes the temporary directory for
parallel configure tests at the end of configure
to avoid checking the already generated modules
while configuring the same Lustre codes with
a different kernel.

Lustre-change: https://review.whamcloud.com/47077
Lustre-commit: TBD (from e0db865abbe8d7fe3a61a649a17f70a88741eb66)

Change-Id: I4505fb932954531d262b5380f17b3df75a80bfa0
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47081
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4080 build: Use -j nproc in lustre dkms install during make

Test-Parameter: trivial

Change-Id: I4f815a0be8900ad427e59cace9a6afc3297710d0
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/47132
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5021 lipe: lpcc_purge recognize pcc pin flag

lpcc_purge check pcc pin flag on lustre file and do not purge
matching cache file in PCC RO.

Change-Id: Iebbe8d1a46a9aefbecd8fd0eb1bbdf71770b0499
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/46920
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5102 utils: Add --pin option for 'lfs pcc attach' command

If --pin option is provided, 'lfs pcc attach' command will
first pin then attach a file. So that user does not need
to run them separately.

Change-Id: I8e06af09abe7922399fd5e8e60132ff24ca31d1a
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47067
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5047 pcc: tunable GID to pin/unpin files

In this patch, we adds a tunable "mdt.*.enable_pin_gid" that
controls the GID to pin/unpin files.
Default should be gid=0, so it is limited to root, gid=-1 so any
user can pin/unpin, or gid=<gid> to allow a specific GID (e.g.
wheel).
We add a virtual "lustre.pin" xattr to set/get the xattr for
non-root users, if mdt.*.enable_pin_gid is set correctly.
Add test case sanity-pcc test_104b to verify it.

Change-Id: I1be92f365cfe8dea278559b2c34fcee8e42d1c4d
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47103
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5128 pcc: avoid to specify attach ID for every pin

In this patch, it avoids the need to specify "-i <attach_id>" for
every pin/unpin operation as in the very common case there is only
a single cache for that client.
If attach ID is not specified, it will select the first dataset
on the client as PCC backend.

Add the test case: sanity-pcc test_205.

Change-Id: I04e2b3d9ca360ae40f3723d7cd9bd78b22602123
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47105
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5113 utils: add pcc pin/unpin commands

Store pin information in xattr trusted.pin.
Add lfs pcc pin/unpin commands to set/clear pcc pin flag.
A pcc-pinned file will not be removed from local storage
cache by lpcc_purge automatically.

Examples of xattr trusted.pin:
trusted.pin=[hsm: 2]
trusted.pin=[hsm: 2,hsm: 3]

Change-Id: I17ddf5ac8dc4eae48c0f6bdd0f2a19240474b0f5
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47083
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5131 build: Add python-*yaml deps for lipe-lpcc

Test-Parameters: trivial

Change-Id: Idd05bd3ae4c12b82392d0b7f2dff826e1954773c
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/47106
Reviewed-by: Raphael Druon <rdruon@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4087 lipe: no lamigo in hot-pools test_59()

In hot-pools test_59(), do not start lamigo. Instead, menaully create
a file with the desired striping (uptodate on fast pool, stale on slow
pool).

Test-Parameters: trivial testlist=hot-pools env=ONLY=59,ONLY_REPEAT=20
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I200b1e890a12d59a0ba5cec5f6450704f35b4df8
Reviewed-on: https://review.whamcloud.com/47031
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-15683 ofd: proper initialize filter_fid in ofd fallocate

Intialize filter_fid buffer and call xattr set XATTR_NAME_FID properly
in ofd_object_fallocate().

Lustre-change: https://review.whamcloud.com/46919
Lustre-commit: TBD (from c77a4635286e5b74034bf6ff65db75bc79e10564)

Fixes: 48457868a02 ("LU-3606 fallocate: Implement fallocate preallocate operation")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ied573c39dde77f935622e9fbedb2d71eb3bd8f5d
Reviewed-on: https://review.whamcloud.com/46924
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-5092 tests: skip hot-pools if yq is not working

Skip hot-pools if yq is not working.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ia8ff851ea7e06626812ca736a0eeec78ca11b930
Reviewed-on: https://review.whamcloud.com/47020
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

EX-3293 tests: use shorter age for lamigo

to speed up the testing

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I433a5ed798c6d4f0f817092a1d21281fab236f78
Reviewed-on: https://review.whamcloud.com/47021
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-5087 lipe: lamigo_job_fini() debugging

In lamigo_job_fini(), only print a debug message for check jobs if the
state of the agent changed.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I6f848d6fc896aea8c45627f9ee542d38bf40d6b4
Reviewed-on: https://review.whamcloud.com/46995
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

LU-14900 tests: do not fail if kmemleak tunable is not writable

Change-Id: Id77430f1e8ff7a8fda6538211a0d36bbe973a889
Test-Parameters: trivial
Fixes: 15c0a21ea9 ("tests: Add kmemleak awareness to test-framework")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46936
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3002 lipe: rename lpurge free{lo,hi} to {max,min}-used

Expressing things in terms of used rather than free is much easier to
understand.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Test-Parameters: trivial testlist=hot-pools
Change-Id: I31ff88db97c0dd1b2ddbaa67aa27e8bec520c7d3
Reviewed-on: https://review.whamcloud.com/46903
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-14644 vvp: wait for nrpages to be updated

truncate_inode_pages() says there still may be a page in a process
of deletion upon return. wait for another thread which is doing
__delete_from_page_cache() to get nrpages updated.

Lustre-change: https://review.whamcloud.com/43464
Lustre-commit: 7d5d004506650c3739898e70d72c9a86b8aeeb88

Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I165b3d0866efaf2eb7e977520ebba4ee831874ab
HPE-bug-id: LUS-8842
Reviewed-on: https://review.whamcloud.com/46954
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-15702 lov: remove lo_trunc_stripeno

Remove the lo_trunc_stripeno member of struct lov_layout_raid0 and add
an lis_trunc_stripe_index array to struct lov_io. This makes the
truncate stripe index information belong to the IO and not to the
concurrently accessed object. This is needed because we do not have
locking that protects it from its initialization in lov_io_iter_init()
to its use in lov_lock_sub_init(). Also remove the unused
lo_write_lock member of struct lov_object.

Fixes: 9801500451 ("LU-14128 lov: correctly set OST obj size")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I799e07059212629fe9d84c5e58c349035a40da9e
Reviewed-on: https://review.whamcloud.com/46953
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

Update lipe version to 2.23.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ie3cdc735a30225dc8f14c3e67292e52057b2170d

RM-620 build: New tag 2.14.0-ddn39

New tag 2.14.0-ddn39

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iddb0af12e9564eb50d5839285e48a9209ab5c030

EX-4333 sec: support supplementary groups from client

The usual way to support more than 2 supplementary groups is to
resort to the server side's identity upcall. This identity upcall
retrieves all user's credentials, including all supplementary groups,
and stores them in cache. But this access to user's credentials from
server side is not always an option.

As an alternative to the server side's identity upcall, we implement
a retry mechanism for intent locking. The client can provide at most
2 supplementary groups in the request sent to the MDS, but sometimes
it does not know which ones are useful for credentials calculation on
server side. For instance in case of lookup, the client does not have
the child inode yet when it sends the intent lock request.
Hopefully, the server can hint at the useful groups, by putting in the
request reply the target inode's GID, and also its ACL. So, in case
the server replies -EACCES, we check the user's credentials against
those, and try again the intent lock request if we find a matching
supplementary group.

On server side, we add a new identity_upcall value named "INTERNAL".
It implements a particular behavior which does not involve an actual
upcall, but instead the cache is filled with supplementary groups read
from the client request, cumulatively at each request.

A problem with the 'runas' utility was found during testing. If no
supplementary group is provided via the '-G' option, then it needs at
least to set the given GID as a supplementary group. Otherwise the
supplementary groups of the invoking user would be silently inherited.
For instance, if root user calls 'runas -u 500 -g 500 $CMD', we must
not execute $CMD with UID:GID 500:500 and supplementary group 0, as
it would make the user executing $CMD part of the superuser group.

Test-Parameters: testgroup=review-dne-part-1 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-part-2 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-selinux-ssk-part-1 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2 env=L_GETIDENTITY=INTERNAL
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4608bb766a70ca12a2142a0e2687813f3a4b9100
Reviewed-on: https://review.whamcloud.com/46493
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13717 sec: fix handling of encrypted file with long name

The ciphertext representation of the name of an encrypted file or
directory can be up to 256 bytes of binary data, if the cleartext
name is up to NAME_MAX. But then this ciphertext is encoded via
critical_encode() before being sent to servers. Once encoded, the
length can exceed NAME_MAX because of the escaped critical
characters.
So make sure ll_prep_md_op_data() accepts those too long encoded names
if it is called for lookup or create of an encrypted file or
directory. In the other cases, the 'name' taken as input is the plain
text version, so it must conform to the NAME_MAX limit.

When carrying out operations on an encrypted file with long name, we
manipulate a digested form whose hash needs to be matched against the
content of the LinkEA. The name found in the LinkEA is not NUL
terminated, so this aspect must be taken care of.

Lustre-change: https://review.whamcloud.com/45163
Lustre-commit: 75414af6bf310244d38284958ecf037d61936726

Fixes: 4d38566a00 ("LU-13717 sec: filename encryption")
Fixes: ed4a625d88 ("LU-13717 sec: filename encryption - digest support")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4b0e51eee5e549ab56292fe0fec3c1be1b487fc7
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46887
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15608 sec: fix DIO for encrypted files

With Direct IO, we do not have proper page cache pages. So we need to
retrieve by ourselves the page mapping and the page index of the page
to be encrypted/decrypted.

For the index, we need to use the offset of the page within the file,
and not the object.
So we rename cl_page's cp_osc_index to cp_page_index for that purpose.
cp_osc_index is redundant with osc_async_page's oap_obj_off and only
used by osc_index(), so we also adapt this function.
cp_page_index is initialized in cl_page_alloc(), and accessed in
the OSC layer where the llcrypt primitives are called.

For the mapping, problem is page->mapping is not set to NULL on page
allocation, so it cannot safely be used to see if a page is a direct
I/O page.
Use cl_page for direct I/O and page->mapping for buffered
I/O. (clpage->cp_inode is only set for direct I/O and
cannot easily be always set.)
Without this, we sometimes get panics when page2inode is
used in the OSC layer. (Note the remaining use in dom is
safe because ll_dom_readpage is a page cache helper and
will never see DIO pages.)

Lustre-commit: 966ca46e4aa2eb39c70e49648ffe6fcaaf475536
Lustre-change: https://review.whamcloud.com/46664/

Fixes: a71e0dd7f7 ("LU-14306 sec: get rid of bad rss-counter state messages")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Icb53a4e45463b8d3febc2e6212b39dc25719d866
Reviewed-on: https://review.whamcloud.com/46819
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14676 lnet: improve hash distribution across CPTs

Change the nid-to-cpt allocation function to use
(sum-by-multiplication of nid bytes) mod (number of CPTs)
to match nid to a CPT. This patch only addresses IPV4 nids.

Make the matching change for the nid-to-cpt function
used by the 'lnetctl cpt-of-nid' utility.

Lustre-change: https://review.whamcloud.com/46233
Lustre-commit: 9b6e27755507b9bb47a1d7b4aede6302a876a14d

Test-parameters: trivial testlist=sanity-lnet

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I1052414947c4cae8c63993ffa21f67cb389bb463
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46398
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14696 llite: check read only mount for setquota

During setting quota, it should fail if the mount is read-only.

Lustre-change: https://review.whamcloud.com/43765
Lustre-commit: 29e00cecc6019fbdb5bd98511970970ac5ef5318

Change-Id: I966ac71d0a4a72dcb998f09ffc0f99ae28498e27
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15637 llite: Fix use of uninitialized fields

We use data from ci_rw to set io_start_index and
io_end_index, which is a problem for mmap because mmap does
not use ci_rw.

When ci_rand_read is set or readahead is disabled, we use
these values to decide how much data to read.

ci_rw is uninitialized, and if the values are non-zero,
we may try to read data beyond the locks we took for our
I/O.

If there is no lock (either because there was never one or
it was cancelled), this results in an LBUG in
osc_req_attr_set when it verifies the pages are covered by
a lock.

Lustre-change: https://review.whamcloud.com/46776/
Lustre-commit: 58fd9b2522325d77643430e361bee5cb625d4bbc (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: If7c8d2eb87a28bf76a6f959e7be7bf636c887cfe
Reviewed-on: https://review.whamcloud.com/46782
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15176 sec: present .fscrypt in subdir mount

fscrypt userspace tool works with a .fscrypt directory at the root of
the file system. In case of subdirectory mount, we virtually present
this .fscrypt directory at the root of the mount point so that fscrypt
can be used. This makes it possible to even do a subdirectory mount of
an encrypted directory, making clients access encrypted content only.
Internally, the .fscrypt directory is always stored at the root of
Lustre.

Lustre-change: https://review.whamcloud.com/46167
Lustre-commit: c12378fba7f0aa2f25de2b295ee110e2932f2a39

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2a0ee360f724da1df49b2be0df986d52e06f45fd
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46888
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14912 obdclass: prefer T10 checksum if the target supports it

If the target actually has T10PI support, we prefer to use that
T10 checksum even it's not the fastest on the client, given
checksum_type is not explicitly set.

Lustre-change: https://review.whamcloud.com/44657
Lustre-commit: 5e9059e08aec6fb36de6788465978de0e962b956

Change-Id: If91217881fcadbc84d1e360e65648344f5ac2447
Test-Parameters: trivial
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/46927
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-15645 obdclass: llog to handle gaps

due to old errors an update llog can contaain gaps in index.
this shouldn't block llog processing and recovery. actual
gaps in transaction sequence should be catched by VBR.

Lustre-change: https://review.whamcloud.com/46837
Lustre-commit: TBD (from b3de0d57bd0f7cd2e918aa9d3f08be1c69697b80)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I11ec817e356f9658118c34706ef3a533e7faba83
Reviewed-on: https://review.whamcloud.com/46884
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15613 utils: add lfs_migrate --cp option

Since "lfs_migrate" is only copying a single file at a time, there is
no real benefit from using "rsync" as the only copytool. Add a short
"--cp" option with suitable arguments to complement "--rsync".

Also allow the copy command to be replaced arbitrarily with RSYNC and
RSYNC_OPTS environment variables to be specified by the caller.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I66c664948d5cc5a7baa5a65550abfaa8d73ebbe5
Reviewed-on: https://review.whamcloud.com/46691
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

DDN-1956 audit: laudit.conf groks mount point

Add a new field to laudit.conf, named "mount", that represents
the mount path of the file system to retrieve audit information
from.

Change-Id: Ic02dc39e9a4e23286916bae92e9ee7a963406e3f
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/46906
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

DDN-1956 audit: scandir's filter returns non-zero to include

When a filter function is used for scandir to select which entries
are to be included, the select routine should return a non-zero
value if the directory entry is to be included.

Fixes: dd3ba5e ("EX-274 lipe: add style check of C language code")
Change-Id: Ic02dc39e9a4e23286916bae92e9ee7a963406e3e
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/46905
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3612 lamigo: use system(2) instead of libssh for local agents

In lamigo, interpret an agent host of '-' to mean the localhost using
system(2) instead of libssh.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I1211f2e43af06659c660fdd57496d09c5d5413fa
Reviewed-on: https://review.whamcloud.com/46902
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4140 lamigo: ignore preallocate status

preallocate status shouldn't be taken into account when space
usage is calculated, otherwise transient errors may cause
incorrect space utilization and unexpected mirrors to that
pool.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7cefd3bc7366bd5b08e1e08cec2eaa4d91e16a90
Reviewed-on: https://review.whamcloud.com/46901
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-15593 mdt: Add option to disable use of SOM

There's currently no way to disable use of strict SOM,
which is a problem if there's ever a SOM bug. This is
tricky to do from the client, but easy on the MDT.

The test just verifies that the size stays the same with
strict SOM disabled, because there's no easy way to check
SOM is disabled unless SOM is broken. (Since it gives the
same value for stat.)

Note this patch requires LU-15609 to be fixed on the client
or the client will see size as 0 when SOM is disabled.

Lustre-change: https://review.whamcloud.com/46683/
Lustre-commit: 2867d497b92f43f06edd7c378828f1fb8c3b300c (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I338fa7e4dd423b07df0f3c5bad4ec6f02e935fea
Reviewed-on: https://review.whamcloud.com/46709
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4965 lipe: use new e2fsprogs xattr API in lipe_scan3

In lipe_scan3, switch to the ext2fs_xattr_handle API since this
handles xattrs stored in a separate file. Update sanity-lipe-scan3
test_108() to verify.

Test-Parameters: trivial testlist=sanity-lipe-find3
Test-Parameters: trivial testlist=sanity-lipe-scan3
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I307d78ba03c6612e5787f71a4682c0af5b23f1c5
Reviewed-on: https://review.whamcloud.com/46875
Tested-by: jenkins <devops@whamcloud.com>

EX-5011 lipe: fixup lipe.spec.in python macros

Update the python macros in lipe.spec.in to support CentOS 8 builds of
the lipe RPMs.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Id910f00a672c4256089fe71d6a98171c1bbb3dd4
Reviewed-on: https://review.whamcloud.com/46891
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5000 lipe: lamigo ssh_channel_read on CentOS 8

Restart ssh_channel_read() after timeout (return 0).

Test-Parameters: env=ONLY=72 trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I6ce5b6e941e432831452a31f8ea840abd4940821
Reviewed-on: https://review.whamcloud.com/46861
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

EX-4999 lamigo: fix argument order in lamigo_show_progress()

Fix argument order in lamigo_show_progress().

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I0f765c149f29b802c365a44d61d6521f05e7504f
Reviewed-on: https://review.whamcloud.com/46860
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

LU-13717 llite: fix crypto patch merge errors

Due to delayed landing of crypto patches, there was a semantic
conflict with another patch that changed the ll_getattr()
function prototype code but did not block the patch from landing.

Test-Parameters: trivial testlist=sanity-sec
Fixes: 3256c354a6 ("LU-13717 sec: filename encryption - symlink support")
Fixes: afec2fef71 ("LU-13717 sec: filename encryption")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia07d6e60f1840889a371538ce5414315e65f6e11
Reviewed-on: https://review.whamcloud.com/46877
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-13587 quota: protect qpi in proc

Access to pool info only when pool is fully inited.
This patch protects from following panic:

    BUG: unable to handle kernel NULL pointer dereference at (null)
    IP: [<ffffffffc0e55e46>] qpi_state_seq_show+0x86/0xe0 [lquota]
    ...
    Call Trace:
    [<ffffffffbbc68b50>] seq_read+0x130/0x440
    [<ffffffffbbcb8380>] proc_reg_read+0x40/0x80
    [<ffffffffbbc4118f>] vfs_read+0x9f/0x170
    [<ffffffffbbc4204f>] SyS_read+0x7f/0xf0
    [<ffffffffbc176ddb>] system_call_fastpath+0x22/0x27

Add test 79 to sanity-quota to check that race between
access to /proc/.../dt-pool_name/info of non-existed pool
with this pool creating doesn't cause a panic.

Lustre-change: https://review.whamcloud.com/43987
Lustre-commit: a11168f46cf01c45a431a804da01cf67e18ecca9

HPE-bug-id: LUS-9938
Change-Id: I8eff846c6c3881a8431a98efb54e660ecb9155bf
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46658
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

LU-13974 llog: check stale osp object

The logic of osp_attr_get has 2 path,
1) return attributes from a cache for health osp object
2) make an out update request and return attributes for stale
osp object, object lose stale state.

When some out update request with llog writes failed, osp object
become stale. But llog handle stay inconsistent (bitmap,count,
last_index), and a next llog_add->llog_osd_write_rec do dt_attr_get,
gets attributes and makes osp object valid, and uses wrong llog
handle data. The result is index jump at llog file - recX, recX+2.
And it makes an error during update log processing if failover take
a place.
The fix adds dt_object_stale function to check osp_object.
llog_osd_write_rec check it and return ESTALE. llog_add would fail
with ESTALE error and doesn't corrupt update log.

Lustre-change: https://review.whamcloud.com/40742
Lustre-commit: 82c6e42d6137f39a1f2394b7bc6e8d600eb36181

HPE-bug-id: LUS-9030
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Iadf53fd816e1c5bde0a19d4c537f0408796c864a
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46802
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15661 nodemap: fix map mode value for 'both'

The patch that introduced the ability to map project IDs with
nodemap changed the value used for the "map both uid and gid"
case, from 0 to 3.
This poses a problem in case of upgrade from a previous Lustre
version, so re-introduce the value 0 as NODEMAP_MAP_BOTH_LEGACY.

Lustre-change: https://review.whamcloud.com/46870
Lustre-commit: TBD (a774ea01a7ffdf177a0229d9794376bddcb9ab57)

Change-Id: I1f605de9c97faff32411da5052e8782a60645767
Fixes: 491f101042 ("LU-14797 sec: add projid to nodemap")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/46869
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4182 sec: support of PCC-RO for encrypted files

In order to support PCC-RO for encrypted files, we decide to store
in PCC the ciphertext version of the Lustre files. We proceed to
decryption of PCC files only in the page cache, so cleartext is just
in memory. When a Lustre file is detached from PCC, or when the
encryption key is removed, we trash those PCC page cache pages.

As PCC files contain ciphertext, their sizes are aligned on
LUSTRE_ENCRYPTION_UNIT_SIZE instead of being lustre inode's clear
text size. In order to keep track of Lustre files' actual sizes,
we use a dedicated xattr on the PCC files. Its value is set at pcc
attach time, which is fine for PCC-RO.

Also add sanity-pcc test_21j to exercise this.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id7d96adb4eb4770c813a042acf7ed6c42224b9bf
Reviewed-on: https://review.whamcloud.com/45910
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15407 test: remove dummy enc key at cleanup

Make sure to remove the dummy encryption key from session keyring
when cleaning up encryption tests.

Lustre-change: https://review.whamcloud.com/46038
Lustre-commit: ec0b308614a2bad18a7a1fd805f36eb8ed6ea5eb

Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I840490fca0a485110d077fe85254ced817fd55e3
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46173
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15406 sec: fix in-kernel fscrypt support

When using in-kernel fscrypt provided by Linux 5.4, the encryption
context can be retrieved by calling the .get_context function defined
in the struct fscrypt_operations of the super_block.
llite needs to retrieve the encryption context explicitly in case of
migration via volatile files.

Lustre-change: https://review.whamcloud.com/45987
Lustre-commit: 2169aed82a32df47be9aef2f249178ede6c7fadd

Fixes: 09c558d16f ("LU-14677 sec: migrate/extend/split on encrypted file")
Fixes: fdbf2ffd41 ("LU-14677 sec: no encryption key migrate/extend/resync/split")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: I76dbd21f0dc95920519ea375c583bc378d7c9f53
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46172
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15176 sec: allow subdir mount of encrypted dir

In case of sub-directory mount of an encrypted directory, we need to
retrieve the encryption context of the root inode of the filesystem.
This is done by making the MDT return this upon getattr reply.

Also add sanity-sec test_60 to exercise this capability.

Lustre-change: https://review.whamcloud.com/45407
Lustre-commit: faf057b46bc770a1a69cacd59e65a40a4b18b9fd

Fixes: 40d91eafe2 ("LU-12275 sec: atomicity of encryption context getting/setting")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ic7a273813533f2904225011b247cdfe995ce9be8
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45909
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15408 sec: confirm encrypted file's hash

It is a good practice to always confirm on server side the encrypted
file's hash included in the digested form sent by the client.

Lustre-change: https://review.whamcloud.com/45964
Lustre-commit: 9d98d1f7739e05bb4decf2614899ccb99b34826c

Fixes: ed4a625d88 ("LU-13717 sec: filename encryption - digest support")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I42212a36b23e4e6e41184a78fa8244c5e2d8dd1f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45965
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14677 sec: remove MIGRATION_ compatibility defines

Remove the MIGRATION_* compatibility flags and use
LLAPI_MIGRATION_* everywhere.

Lustre-change: https://review.whamcloud.com/44957
Lustre-commit: e42d2d67d3a0dcc726d1424d3158b6f649b5abd7

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iab2a2f6dfc435377e9db0d4963547841b2cbc403
Reviewed-on: https://review.whamcloud.com/45908
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14677 sec: no encryption key migrate/extend/resync/split

Allow some layout operations on encrypted files, even when the
encryption key is not available:
- lfs migrate
- lfs mirror extend
- lfs mirror resync
- lfs mirror verify
- lfs mirror split
We allow these access patterns to applications that know what they are
doing, by using the specific flag O_FILE_ENC and O_DIRECT.

Also add sanity-sec test_59a,b,c to exercise these access patterns.

Lustre-change: https://review.whamcloud.com/44024
Lustre-commit: fdbf2ffd41fa5660782d5ca8489ec2eb644c8113

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ieaeee0e5bf7643f18d775fe6daa5e31c2f349f8c
Reviewed-on: https://review.whamcloud.com/44182
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13717 sec: missing defs in includes for client encryption

Add a few missing definitions in lustre_crypto.h that are required
in case Lustre client encryption is built against the in-kernel
fscrypt library.

Lustre-change: https://review.whamcloud.com/45221
Lustre-commit: 79ac7c144539e0d964db329c341ebf30a8472f5c

Fixes: d08ae042d8 ("LU-13717 sec: rework includes for client encryption")
Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1965503554dcf660770d201444cfafd54aa84dce
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45734
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15027 sec: initialize ll_inode_info for fake inode

When creating an encrypted symlink, we need to make use of a fake
inode in order to be able to encrypt the target name before sending
the create request to the MDS.
This fake inode needs minimal initialization, but it is at least
required to properly initialize the ll_inode_info associated with this
fake inode.

Lustre-change: https://review.whamcloud.com/45023
Lustre-commit: 3fb7b6271855c0b12c5a560c7f6287cdda3d1cd6

Fixes: e735298935 ("LU-13717 sec: filename encryption - symlink support")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I20c30d873f9bffdbdc8b5f272cb8b80e5be7fbfb
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45733
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13717 sec: filename encryption - symlink support

On client side, call the appropriate llcrypt primitives from llite,
to proceed with symlink encryption before sending requests to servers
and symlink decryption upon request receipt.
The tricky part is that llcrypt needs an inode to encrypt the target
name. But by the time we prepare the symlink creation request to be
sent to the server with the target name (in ll_new_node), we do not
have an inode yet (it will be obtained only after we get the server
reply). So we create a fake inode and associate the right encryption
context to it, so that the symlink gets encrypted properly.

In order to report the correct size for an encrypted symlink (which is
ought to be the length of the symlink target), we need to read the
symlink target and decrypt or decode it in ->getattr(). This has a
performance hit, but given that the symlink target is cached in
->i_link (when the key is available), the symlink will not have to be
read and decrypted again later when it is actually followed,
readlink() is called, or lstat() is called again.
This part of the patch is adapted from kernel commit
d18760560593e5af921f51a8c9b64b6109d634c2
"fscrypt: add fscrypt_symlink_getattr() for computing st_size"

With encrypted file names, a symlink target is binary. So make sure
server side can handle that, by switching sp_symname to a
struct lu_name in struct md_op_spec.

Lustre-change: https://review.whamcloud.com/43394
Lustre-commit: e735298935b64541fc561bd9e978cd7af48c503e

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ic6892fca8926a35001697c54aaf05d15563b139d
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45732
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13717 sec: filename encryption - digest support

A number of operations are allowed on encrypted files without the key:
- read file metadata (stat);
- list directories;
- remove files and directories.
In order to present valid names to users, cipher text names are base64
encoded if they are short. Otherwise we compute a digested form of the
cipher text, made of the FID (16 bytes) followed by the second-to-last
cipher block (16 bytes), and we base64 encode this digested form for
presentation to user.
These transformations are carried out in the specific overlay
functions, that now need to know the fid of the file.

As the digested form does not contain the whole cipher text name,
server side needs to proceed to an operation by FID for requests such
as lookup and getattr. It also relies on the content of the LinkEA to
verify the digested form as received from client side.

Lustre-change: https://review.whamcloud.com/43392
Lustre-commit: ed4a625d88567a2498c3fe32fd340ae7985e6ad0

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I45d10a426373c2cfe0b92a58c351da452d085d7d
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45731
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13717 sec: filename encryption

On client side, call the appropriate llcrypt primitives from llite,
to proceed with filename encryption before sending requests to servers
and filename decryption upon request receipt.
Note we need specific overlay functions to handle encoding and
decoding of encrypted filenames, as we do not want server side to deal
with binary names before they reach the backend file system layer.

On server side, mainly the OSD layer, we need to know the encryption
status of files being processed.
If an object belongs to an encrypted file, the filename has been
encoded by the client because it is binary, so it needs to be decoded
before being handed over to the backend file system layer.
And conversely, the filename of an encrypted file has to be encoded
before being sent over the wire.
Note server side is osd-ldiskfs only for now.

Lustre-change: https://review.whamcloud.com/43390
Lustre-commit: 4d38566a004f6a636c37ec0c86f053be9b905bd7

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I7ac9047f5a046b8bc63afdbbb1f28e78aa5c8c7e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45730
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13717 sec: make client encryption compatible with ext4

In order to benefit from encrypted file handling implemented in
e2fsprogs, we need to adjust the way Lustre deals with encryption
context of files.

First, the encryption context needs to be stored in an xattr named
"encryption.c" instead of "security.c". But neither llite nor ldiskfs
has an xattr handler for this "encryption." xattr type. So we need
to export ldiskfs_xattr_get and ldiskfs_xattr_set_handle symbols for
this to work.

Second, we set the LDISKFS_ENCRYPT_FL flag on files for which we set
the 'encryption.c' xattr. But we just keep this flag for on-disk
inodes, and make sure the flag is cleared for in-memory inodes.
The purpose is to help e2fsprogs with encrypted files handling, while
not disturbing Lustre server side with the encryption flag (servers
are not supposed to know about it for Lustre client-side encryption).

To maintain compatibility with 2.14 in which encryption context is
stored in "security.c" xattr, we try to fetch enc context from this
xattr if getting it from "encryption.c" fails. On client side, in all
cases everything looks like encryption context is stored in
"encryption.c".

Lustre-change: https://review.whamcloud.com/45211
Lustre-commit: 4231fab66eab3e984499bf0c6bd4514692a409fa

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I784ec530f0dfdd2743169ba2326ff6c5cdd4e85a
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/45766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4811 lipe: Allows nfs/nfs4 targets for loris

Loris only supprots local filesystems (e.g. ext4, xfs and brtfs) as
backup target.

embedded MDS (ESx00X) doesn't have enough space in the local disk.
it needs to eliminate filesystem type and allow NFS/NFS4 for
loris's backup targets.

Test-Parameters: trivial
Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Change-Id: I010793402065cd553c18a7f408240841e4a22565
Reviewed-on: https://review.whamcloud.com/46515
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4865 lipe: add timestamps to lamigo and lpurge log messages

Added optional timestamp to log messages; Added command line
option --timestamps to lamigo/lpurge and 'timestamps' keyword
to lpurge config. Added timestamps to log files to
hot-pools test frame by default.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: If58f20eed31c6df30a70f0cb1c93430040f8fe62
Reviewed-on: https://review.whamcloud.com/46599
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

EX-4863 lipe: initialize cmd in lamigo_dump_jobs()

Fix an uninitialized use of cmd in lamigo_dump_jobs().

Test-Parameters: trivial testlist=hot-pools
Fixes: 8a29c986eb ("EX-4103 lamigo: rename some "check" functions")
Change-Id: I79c32d9b32a0a06fbd72c0b6dd21d8c6919aebc6
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46554
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4963 lipe: rescan after registering a new changelog user

In lamigo, if we register a new changelog user then force a rescan of
the device. Add hot-pools test_6c() to verify. Adjust hot-pools to be
more precise about param matching.

Test-Parameters: trivial testlist=hot-pools
Test-Parameters: trivial testlist=hot-pools env=ONLY=6c,ONLY_REPEAT=20
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I9576d50f87ffe0eb5e44c62087fca317ddce50c7
Reviewed-on: https://review.whamcloud.com/46781
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

EX-4965 lipe: use new e2fsprogs xattr API

Switch to the ext2fs_xattr_handle API since this handles xattrs stored
in a separate file. Add sanity-lipe test_100() to verify.

Test-Parameters: testlist=sanity-lipe
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ibc39d01ecfb04307755f44751fc69ffbd79b5d69
Reviewed-on: https://review.whamcloud.com/46784
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4103 lamigo: rename some "check" functions

Despite the name of the feature, files are "hot" or "cold", while
pools are "fast" or "slow". Rename and reorganize some functions with
this in mind:

  lamigo_check_hot -> lamigo_sync_hot_files
  lamigo_new_job_for_hot -> lamigo_submit_sync
  lamigo_check_hot_one -> lamigo_sync_hot_to_fast
  lamigo_check_hot_on_cold -> lamigo_sync_hot_to_slow
  lamigo_get_hot -> lamigo_get_hot_files

Test-Parameters: trivial testlist=hot-pools
Change-Id: I2833c8828d73e50a72db8a19aae16d1400eccd66
Reviewed-on: https://review.whamcloud.com/45410
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4539 lipe: add lipe_find3 manpage

Add a manpage for lipe_find3. Add some undistributed internal notes
for posterity.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I2291b7b8e8d4b8706d4dc5a3aec8b4815acfff10
Reviewed-on: https://review.whamcloud.com/46731
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14587 ptlrpc: remove LASSERT in nrs_polices proc handler

It's not necessary to LASSERT() in nrs_polices proc handler.
CERROR() and returning error is good enough.

Lustre-change: https://review.whamcloud.com/45200
Lustre-commit: 9997f94d4b6ee335d2bf86f94bd43464d5b8f061

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: I09f06dc4ab90e49b2df66a9b47a74678c64cdd2f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/46769
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15615 target: Free t10pi crypto state on error

Looks like when error happens we forgot to release crypto state that
not only leaks memory directly, but potentially can tie in-memory
pages too.

Change-Id: Ia0870ccbb194e4e9ca8701e1c01d519745c236df
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46761
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>

LU-15263 quota: fix bug in qmt_pool_recalc

env should be freed at the end of qmt_pool_recalc,
as it is needed in qpi_putref. It causes a panic,
if pool has the last reference:
    BUG: unable to handle NULL pointer dereference at 000000000000a0
    IP: lu_context_key_get+0x17/0x30 [obdclass]
    Call Trace:
      lu_object_free.isra.30+0x68/0x170 [obdclass]
      lu_object_put+0xc5/0x3e0 [obdclass]
      qmt_pool_free+0x30c/0x590 [lquota]
      qmt_pool_recalc+0x365/0x1260 [lquota]
      kthread+0xd1/0xe0
      ret_from_fork_nospec_begin+0x21/0x21

Lustre-change: https://review.whamcloud.com/45632
Lustre-commit: 57d88137e12472cf5ea08aa28957b4767abd475c

HPE-bug-id: LUS-10426
Change-Id: Ic23dcb858ff811757f38948aa572c936c076e21e
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-on: https://review.whamcloud.com/46794
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13756 quota: up_read leak in qmt_pool_lookup

qmt_pool_lock is not released if qti_pools_add fails in
qmt_pool_lookup.

Lustre-change: https://review.whamcloud.com/45106
Lustre-commit: d16b3141119a3b75276914ad3601e0dd27579b2b

Change-Id: Ic2adb44468d51af7aefcbb91279260ae6f85d67a
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/46793
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15079 quota: include qsd_thread_info into mgs thread

mgs service thread envs do not get supplied with qsd_thread_info,
which may lead to the failure shown below:
    (lu_object.h:1274:lu_env_info()) ASSERTION( info ) failed:
    Pid: 146951, comm: ll_mgs_0003 3.10.0-957.1.3957.1.3.x4.3.25
    Call Trace:
      libcfs_call_trace+0x8e/0xf0 [libcfs]
      lbug_with_loc+0x4c/0xa0 [libcfs]
      qsd_refresh_usage+0x25e/0x2f0 [lquota]
      qsd_op_adjust+0x2f1/0x730 [lquota]
      osd_object_delete+0x2b2/0x360 [osd_ldiskfs]
      lu_object_free.isra.32+0x68/0x170 [obdclass]
      lu_site_purge_objects+0x2fe/0x530 [obdclass]
      lu_object_find_at+0x371/0xa60 [obdclass]
      dt_locate_at+0x1d/0xb0 [obdclass]
      llog_osd_open+0x50e/0xf30 [obdclass]
      llog_open+0x15a/0x3e0 [obdclass]
      llog_origin_handle_open+0x334/0x720 [ptlrpc]
      tgt_llog_open+0x33/0xe0 [ptlrpc]
      mgs_llog_open+0x46/0x460 [mgs]
      tgt_request_handle+0x96a/0x1680 [ptlrpc]

Supply msg service context with qsd_thread_info.

Lustre-change: https://review.whamcloud.com/45181
Lustre-commit: 69a9042f26fa22b1d5b2ad7b3cb8024d508268dd

Change-Id: If8664b81e1f64df015dad46ba26c9c1d1e3f54bf
HPE-bug-id: LUS-10334
Signed-off-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/46792
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>