Whamcloud - gitweb
fs/lustre-release.git
4 months agoLU-16023 tests: sanity-quota/8 should return success 66/47966/4
Alex Zhuravlev [Mon, 18 Jul 2022 10:28:49 +0000 (13:28 +0300)]
LU-16023 tests: sanity-quota/8 should return success

sanity-quota/8 should return success explicitly

Test-Parameters: trivial testlist=sanity-quota
Fixes: bc69a8d058 ("LU-8621 utils: cmd help to stdout or short cmd error")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Id30bfd3e0bafedb6516471accbc0519cc640d2bd
Reviewed-on: https://review.whamcloud.com/47966
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15282 tests: relax test_51d thresholds somewhat 78/47978/3
Andreas Dilger [Mon, 18 Jul 2022 22:12:12 +0000 (16:12 -0600)]
LU-15282 tests: relax test_51d thresholds somewhat

Added combinations for sanity.sh test_51d are failing some fraction
of the time.  Relax thresholds somewhat to avoid spurious failures,
while keeping added configs to detect major regressions.

Test-Parameters: trivial
Fixes: fd5c915eff ("LU-15282 tests: improve sanity test_51d coverage")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2d28f77fa377a1eb96a00cf35827b9ebc5af806b
Reviewed-on: https://review.whamcloud.com/47978
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15983 lnet: Define KFILND network type 30/47830/5
Chris Horn [Wed, 29 Jun 2022 00:24:32 +0000 (19:24 -0500)]
LU-15983 lnet: Define KFILND network type

Define the KFILND network type. This reserves the network type number
for future implementation and allows creation of kfi peers and
adding routes to kfi peers.

Test-Parameters: trivial testlist=sanity-lnet
HPE-bug-id: LUS-11060
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I9111645f1290c8af4937d1b2689a068df81922a4
Reviewed-on: https://review.whamcloud.com/47830
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15850 mdt: pack default LMV in open reply 76/47576/7
Lai Siyao [Thu, 9 Jun 2022 11:26:40 +0000 (07:26 -0400)]
LU-15850 mdt: pack default LMV in open reply

Add flag MDS_OPEN_DEFAULT_LMV to indicate that default LMV should be
packed in open reply, otherwise if open fetches LOOKUP lock, client
won't know directory has default LMV, and in subdir creation default
LMV won't take effect.

Test-Parameters: clientversion=2.14 testlist=sanity mdtcount=4 mdscount=2 env=SANITY_EXCEPT="39l 134b 150b 160g 205a 208 230e 230p 270a 300g 807"
Test-Parameters: serverversion=2.14 testlist=sanity mdtcount=4 mdscount=2 env=SANITY_EXCEPT="65n 247f"
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: If2300ca39f406169eff9eab8f973ca1c2bfc8202
Reviewed-on: https://review.whamcloud.com/47576
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-14472 quota: skip non-exist or inact tgt for lfs_quota 71/41771/19
Hongchao Zhang [Wed, 15 Dec 2021 12:11:17 +0000 (20:11 +0800)]
LU-14472 quota: skip non-exist or inact tgt for lfs_quota

The nonexistent or inactive targets (MDC or OSC) should be skipped
for "lfs quota".

Change-Id: I25eece413715e4e05dd94ccbfd101220da7477f9
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41771
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-11407 tgt: cleanup job_stats output printing 64/37764/24
Andreas Dilger [Fri, 10 Jan 2020 21:18:53 +0000 (14:18 -0700)]
LU-11407 tgt: cleanup job_stats output printing

Escape non-printable and other special characters in the JobID
name, which may be passed from the client environment, to avoid
breaking YAML format parsing.  We can't use the kernel "%*pE"
escape format, because that doesn't have any option to escape
printable characters like quotes or regular spaces.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If6a0bc4276fae03305f94e8c85d8f109913ebbe5
Reviewed-on: https://review.whamcloud.com/37764
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-6612 utils: strengthen llog_reader vs wrong format/header 54/15654/5
Bruno Faccini [Mon, 20 Jul 2015 14:30:11 +0000 (16:30 +0200)]
LU-6612 utils: strengthen llog_reader vs wrong format/header

The following snippet shows that llog_reader can be puzzled due to
an invalid 0 for the number of records when parsing an expected
LLOG file header :
root# dd if=/dev/zero bs=4096 count=1 of=/tmp/zeroes
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000263962 s, 15.5 MB/s
root# llog_reader /tmp/zeroes
Memory Alloc for recs_buf error.
Could not pack buffer; rc=-12

Test-Parameters: trivial testlist=sanity,sanity-hsm
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I12be79e6c6a5da384a5fd81878a76a7ea8aa5834
Reviewed-on: https://review.whamcloud.com/15654
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15984 o2iblnd: debug message is missing a newline 33/47933/2
Serguei Smirnov [Mon, 11 Jul 2022 22:49:04 +0000 (15:49 -0700)]
LU-15984 o2iblnd: debug message is missing a newline

Add missing newline to one of the debug messages in
kiblnd_pool_alloc_node.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I541622322ea6166892270dbfd1567cc64f8c314c
Reviewed-on: https://review.whamcloud.com/47933
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15938 lod: prevent endless retry in recovery thread 98/47698/3
Mikhail Pershin [Wed, 22 Jun 2022 10:27:48 +0000 (13:27 +0300)]
LU-15938 lod: prevent endless retry in recovery thread

- abort lod_sub_recovery_thread() by obd_abort_recov_mdt in
  addition to obd_abort_recovery
- handle 'short llog' situation gracefully, when remote llog
  is shorter than local copy header expects, trust remote llog
  data and consider llog processing as finished
- on other errors during remote llog read, set obd_abort_recov_mdt
  but not obd_abort_recovery in attempt to skip MDT-MDT recovery
  only and continue with client recovery while possible
- fix parsing problem with 'abort_recov' and 'abort_recov_mdt' in
  lmd_parse() causing no MDT recovery abort but client recovery
  abort always. Allow also 'abort_recovery_mdt' mount option name

The original case with endless retry is caused by such de-sync
between local llog structures and remote llog. The local llog
header says there is record with some ID, so recovery thread
is trying to get that record from remote llog. Meanwhile there
is no such record on remote server, so it reads whole llog and
return it back properly but llog processing consider that as
incomplete llog due to network issues and retry endlessly.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib127fd0d1abd5289d90c7b4b3ca74ab6fc78bc71
Reviewed-on: https://review.whamcloud.com/47698
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-10994 clio: Remove cl_2queue_add wrapper 51/47651/4
Shivani Bhardwaj [Mon, 27 Jun 2022 15:13:25 +0000 (11:13 -0400)]
LU-10994 clio: Remove cl_2queue_add wrapper

Remove the wrapper function cl_2queue_add() and replace all its calls in
different files with the function it wrapped. Also, comments are added
wherever necessary to make the working of function clear. Prototype of
the function is also removed from the header file as it is no longer
needed.

Linux-commit: 53f1a12768a55e53b2c40e00a8804b1edfa739b3

Change-Id: Ic746c45e3dda9fdf3f1d2f8c6204d80fec5c058f
Signed-off-by: Shivani Bhardwaj <shivanib134@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/47651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15925 lnet: add debug messages for IB 83/47583/4
Cyril Bordage [Thu, 9 Jun 2022 21:41:54 +0000 (23:41 +0200)]
LU-15925 lnet: add debug messages for IB

If net debug is enabled, information about connection, when
tx status is ECONNABORTED, is collected (only for IB).

Test-Parameters: trivial
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Change-Id: I44a33703931630b85cc0e847e2a038217b7967c6
Reviewed-on: https://review.whamcloud.com/47583
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15902 obdclass: dt_try_as_dir() check dir exists 83/47483/7
Lai Siyao [Thu, 19 May 2022 22:31:07 +0000 (18:31 -0400)]
LU-15902 obdclass: dt_try_as_dir() check dir exists

If an object is not directory, but dt_lookup() is called on it, it
may crash because .do_lookup is NULL for non-directory file.

Add argument to check object existence and type in dt_try_as_dir(),
and for object to create, skip this check.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I51df0cbb5a4e7abca370ee27dac678f995b76159
Reviewed-on: https://review.whamcloud.com/47483
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15880 quota: fix issues in reserving quota 25/47425/4
Hongchao Zhang [Fri, 24 Jun 2022 14:06:08 +0000 (22:06 +0800)]
LU-15880 quota: fix issues in reserving quota

Calling "chgrp" with unprivileged user will reserve quota space
before changing the GID of the file, and the reserved quota space
will be freed after its transaction is committed. there are some
issues in the current implementation,
1, the reserved quota isn't freed in case of error in "mdd_attr_set"
   and "tgt_cb_last_committed".
2, during freeing the reserved quota, the quota space to free is
   set as the same parameter as reserving the quota, which could
   be wrong, for instance, the reserving quota space will be 0 if
   the corresponding quota ID isn't enforces, but the call will
   return without error.

Like the "qsd_op_begin/qsd_op_end",The patch also adds reference to
the lquota_entry gotten during reserving quota and release it during
freeing the reserved quota to prevent potential issue.

Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I098cde7d5e89fe8b9eaab0ae4bc285a4ac6c2281
Reviewed-on: https://review.whamcloud.com/47425
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15991 kernel: kernel update RHEL7.9 [3.10.0-1160.71.1.el7] 65/47865/2
Jian Yu [Tue, 5 Jul 2022 06:08:40 +0000 (23:08 -0700)]
LU-15991 kernel: kernel update RHEL7.9 [3.10.0-1160.71.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.71.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I89215145ea8da2925e5c8c01cdf963ba8a087877
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-16000 utils: align updatelog parameters in llog_reader 13/47913/2
Etienne AUJAMES [Fri, 8 Jul 2022 10:36:10 +0000 (12:36 +0200)]
LU-16000 utils: align updatelog parameters in llog_reader

Parameters in update log records are aligned on 64bits. llog_reader
do not aligned these parameters: if a parameters size is not mutiple
of 8, the next parameter size will be read incorrectly.

Test-Parameters: trivial
Fixes: 9962d6f ("LU-14617 utils: llog_reader updatelog support")
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I6871614ab4ea79d59c3c3b4644b377de395bad56
Reviewed-on: https://review.whamcloud.com/47913
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15993 ofd: don't leak pages if nodemap fails 73/47873/4
Alex Zhuravlev [Tue, 5 Jul 2022 11:24:03 +0000 (14:24 +0300)]
LU-15993 ofd: don't leak pages if nodemap fails

ofd_commitrw() shouldn't exit w/o calling ofd_commitrw_write(),
otherwise the pages taken in ofd_preprw() are leaked.

same in mdt_obd_commitrw()

Fixes: bbfdc7c167 ("LU-14739 quota: fix quota with root squash enabled")

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Icd60c7ab80c5a7b65603d7da0d2e83872dc6b97f
Reviewed-on: https://review.whamcloud.com/47873
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-8582 target: send error reply on wrong opcode 61/47761/6
Li Xi [Tue, 21 Jun 2022 12:06:22 +0000 (20:06 +0800)]
LU-8582 target: send error reply on wrong opcode

Unknown opcode does not necessarily means insane client. A new client
might send RPCs with new opcodes to an old server. The client might
desperately stuck there waiting for a reply. So, send an error back
when RPC has a wrong opcode.

This patch returns the EOPNOTSUPP to client instead of block. ENOTSUPP
is not used here since strerror() does not understand ENOTSUPP.

OBD_FAIL_OST_OPCODE=0x253 is added for fault injection test of opcode.
To test whether an invalid opcode is handled properly on OST, use the
following command:

lctl set_param fail_val=${opcode} fail_loc=0x253

Change-Id: I46ca62bc532b92368e06a4f883b102c7186c453c
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/47761
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15888 build: Debian dkms-debs requires ed and libkeyutils 55/47455/7
Shaun Tancheff [Fri, 8 Jul 2022 13:42:00 +0000 (08:42 -0500)]
LU-15888 build: Debian dkms-debs requires ed and libkeyutils

dkms install/build needs dependencies on libmount-dev,
libkeyutils1, and libkeyutils-dev

Debian does not install the 'ed' package by default.
Without the 'ed' package the version is not correctly added
to the changelog and parsed to the package names.

Debian does not have linux-image or linux-headers psuedo
packages so require the arch specific ones, ex:
   linux-image | linux-image-amd64 | linux-image-arm64
and:
   linux-headers | linux-headers-amd64 | linux-headers-arm64
respectively.

o2ib fails to find Debian in-kernel Module.symvers and
should check $LINUX_OBJ/Module.symvers before failing.

HPE-bug-id: LUS-10984
Test-Parameters: trivial
Fixes: 85a6eebeca1 ("LU-15652 build: On Debian detect -common kernel headers")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I82e2689f3af4b9ce106ee3ab6b4109d2709c8872
Reviewed-on: https://review.whamcloud.com/47455
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15886 lfsck: remove unreasonable assertions 47/47447/6
Lai Siyao [Wed, 18 May 2022 22:21:52 +0000 (18:21 -0400)]
LU-15886 lfsck: remove unreasonable assertions

Remove unreasonable assertions in LFSCK code:
* lfsck->li_obj_dir and lfsck->li_lmv may be NULL if object wan't
  initialized successfully.
* orphan objects under ldiskfs /lost+found may not exist.
* object may not be directory in lfsck_verify_lpf()->
  lfsck_verify_linkea().
* for corner case
  (leh->leh_reccount == 0 && leh->leh_overflow_time != 0),
  LASSERT(ldata->ld_leh->leh_reccount > 0) will be triggerred in
  lfsck_namespace_linkea_clear_overflow(), remove this assertion,
  and this corner case can be handled correctly in current lfsck
  code.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: If114c7882a2c083e83fcfac5981eddfa526d1426
Reviewed-on: https://review.whamcloud.com/47447
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15868 lfsck: don't crash upon dir migration failure 81/47381/7
Lai Siyao [Tue, 17 May 2022 11:11:25 +0000 (07:11 -0400)]
LU-15868 lfsck: don't crash upon dir migration failure

LFSCK against directories that were migrated, but failed may crash,
it's because lost+found directory may not be initialized correctly,
and this error is skipped on purpose, add check in code that
dereference it.

lfsck_verify_lpf() may dereference NULL "child2".

lmv_name_to_stripe_index() should support stripe LMV, which is used
by LFSCK to verify name hash.

Add OBD_FAIL_OUT_EIO to simulate sub transaction failure.

Add sanity-lfsck 15d to verify LFSCK won't crash upon directory
migration failure.

Update sanity-lfsck 4 and 5 to start mds1 with OI scrub enabled, and
wait for mds1 OI scrub finish, otherwise LFSCK may fail to verify
lost+found later.

Test-Parameters: mdscount=2 mdtcount=4 testlist=sanity-lfsck \
env=ONLY=15d,ONLY_REPEAT=100
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I1b1872da2b4ef8f7403effc4d1d3e298c6a0b7e6
Reviewed-on: https://review.whamcloud.com/47381
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15626 tests: Fix "error" reported by shellcheck for recovery-random-scale 66/46866/8
Arshad Hussain [Fri, 18 Mar 2022 07:19:24 +0000 (12:49 +0530)]
LU-15626 tests: Fix "error" reported by shellcheck for recovery-random-scale

This patch fixes "error" issues reported by shellcheck
for file lustre/tests/recovery-random-scale.sh. This
patch also moves spaces to tabs.

Test-Parameters: trivial clientcount=6 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180,SLOW=no testlist=recovery-random-scale
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I9e7c6fcd726d5fc2141a7ced73ea78ee6a43ec22
Reviewed-on: https://review.whamcloud.com/46866
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15626 tests: Fix "error" reported by shellcheck for recovery-mds-scale 65/46865/12
Arshad Hussain [Fri, 18 Mar 2022 07:25:25 +0000 (12:55 +0530)]
LU-15626 tests: Fix "error" reported by shellcheck for recovery-mds-scale

This patch fixes "error" issues reported by shellcheck
for file lustre/tests/recovery-mds-scale.sh. This patch
also moves spaces to tabs.

Test-Parameters: trivial clientcount=6 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180,DURATION=600,SLOW=no testlist=recovery-mds-scale
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I6c098809835950e1f781e04a6898895592407948
Reviewed-on: https://review.whamcloud.com/46865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15132 hsm: Protect against parallel HSM restore requests 67/45367/15
Etienne AUJAMES [Thu, 21 Oct 2021 14:31:01 +0000 (16:31 +0200)]
LU-15132 hsm: Protect against parallel HSM restore requests

Multiple parallel accesses (read/write) to the same released file
could cause multiple HSM restore requests to be sent.
On the MDT side, each restore request waits the first one to complete
before grabbing the MDS_INODELOCK_LAYOUT LCK_EX and registering the
llog record.

This could cause several MDT threads to hang for the same restore
request sent in parallel. In the worst case, all MDT threads can
hang and the MDS is not longer able to handle requests.

This patch checks if an HSM restore handle exists before taking the
lock.

Test-Parameters: testlist=sanity-hsm,sanity-hsm
Test-Parameters: testlist=sanity-hsm env=ONLY=12s,ONLY_REPEAT=50
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I9584edc2c7411aa41b2e318e55f57c117d1c3dfb
Reviewed-on: https://review.whamcloud.com/45367
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15913 mdt: disable parallel rename for striped dirs 93/47593/9
Andreas Dilger [Sat, 11 Jun 2022 01:47:00 +0000 (19:47 -0600)]
LU-15913 mdt: disable parallel rename for striped dirs

Parallel rename should not be done within striped directories to
avoid remote updates.  These are like cross-directory renames.

Add tunables for parallel directory rename in case of problems.
These can be configured separately for files and directories.

    mdt.*.enable_parallel_rename_dir
    mdt.*.enable_parallel_rename_file

Fixes: 90979ab390 ("LU-12125 mds: allow parallel directory rename")
Fixes: d76cc65d5d ("LU-12125 mds: allow parallel regular file rename")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I384976cd1c9f401169336ee7a479ba0e3dd9f4ee
Reviewed-on: https://review.whamcloud.com/47593
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15521 spec: fix more bare words error with rpm 4.16 33/47833/2
Jian Yu [Wed, 29 Jun 2022 21:23:59 +0000 (14:23 -0700)]
LU-15521 spec: fix more bare words error with rpm 4.16

This patch fixes more bare words errors and extra tokens
warnings with rpm 4.16.

Test-Parameters: trivial

Change-Id: Ic40b5763d1cb362d5aa77b06e9a5768b2abbc708
Fixes: 597a6bf9e085 ("LU-15521 spec: fix bare words error with rpm 4.16")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47833
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15653 client: able to cleanup devices manually 59/46859/7
Mikhail Pershin [Tue, 22 Feb 2022 17:34:37 +0000 (20:34 +0300)]
LU-15653 client: able to cleanup devices manually

Using 'lctl cleanup/detach' could be needed in situations
with unclean umount. Meanwhile that doesn't work now for
LMV and also could cause panic after all

Patch restores ability to cleanup/detach client devices
manually.
- debugfs and lprocfs cleanup in lmv_precleanup() are moved
  lmv_cleanup() to be not cleared too early. This prevents
  hang on 'lctl cleanup' for LMV device
- test 172 is added in sanity. It skips device cleanup during
  normal umount, keeping device alive without client mount
  then manually cleanups/detaches them
- prevent negative lov_connections in lov_disconnect() and
  handle it gracefully
- remove obd_cleanup_client_import() in mdc_precleanup(),
  it is called already inside osc_precleanup_common()

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I8a3868fabd1d805e827d04852d1614a3fe57ce35
Reviewed-on: https://review.whamcloud.com/46859
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15894 ofd: revert range locking in ofd 66/47466/4
Andrew Perepechko [Sun, 24 Apr 2022 14:48:09 +0000 (17:48 +0300)]
LU-15894 ofd: revert range locking in ofd

After commit 301d76a711 (LU-14876), range locking is no longer needed
in ofd, because 301d76a711 itself prevents the original data corruption
fixed by range locking. At the same time, range locking in ofd adds
unnecessary overhead, we can even see serialization under specific load.

This patch reverts range locking but keeps recovery-small test 148
to test the original corruption scenario case.

Change-Id: Ic795bcfb1e249c4927f66b6bad456f5511819861
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
HPE-bug-id: LUS-9890
Fixes: 35679a730 ("LU-10958 ofd: data corruption due to RPC reordering")
Reviewed-on: https://review.whamcloud.com/47466
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15759 libcfs: debugfs file_operation should have an owner 35/47335/7
Mr NeilBrown [Fri, 13 May 2022 02:16:12 +0000 (12:16 +1000)]
LU-15759 libcfs: debugfs file_operation should have an owner

If debugfs a file is open when unloading the libcfs/lnet module, it
produces a kernel Oops (debugfs file_operations callbacks no longer
exist).

Crash generated with routerstat (/sys/kernel/debug/lnet/stats):
[ 1449.750396] IP: [<ffffffffab24e093>] SyS_lseek+0x83/0x100
[ 1449.750412] PGD 9fa14067 PUD 9fa16067 PMD d4e5d067 PTE 0
[ 1449.750428] Oops: 0000 [#1] SMP
[ 1449.750883]  [<ffffffffab7aaf92>] system_call_fastpath+0x25/0x2a
[ 1449.750897]  [<ffffffffab7aaed5>] ?
system_call_after_swapgs+0xa2/0x13a

This patch adds an owner to debugfs file_operation for libcfs and
lnet_router entries (/sys/kernel/debug/lnet/*).

The following behavior is expected:
$ modprobe lustre
$ routerstat 10 > /dev/null &
$ lustre_rmmod
rmmod: ERROR: Module lnet is in use
Can't read statfile (ENODEV)
[1]+  Exit 1                  routerstat 10 > /dev/null
$ lustre_rmmod

Note that the allocated 'struct file_operations' cannot be freed until
the module_exit() function is called, as files could still be open
until then.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia0920313e0c2a4b6cdc875fed08221e174a12a73
Reviewed-on: https://review.whamcloud.com/47335
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15779 ofd: don't hold read lock over bulk 26/47126/5
Alex Zhuravlev [Sun, 24 Apr 2022 11:27:18 +0000 (14:27 +0300)]
LU-15779 ofd: don't hold read lock over bulk

as this can block all operations on OST:

1) ofd_preprw_read() takes a shared object lock and initiates BULK
2) OUT needs an exclusive object lock on the same object
3) ofd_commitrw_write() starts transaction and now has to wait
   for OUT to get and release that exclusive object lock (step 2)
4) number of threads can get stuck waiting for ofd_commit_write()
   to stop it's transaction

this patch drops a shared object lock before BULK transfer.
at the moment it's not clear how such read would race with
object removal on ZFS - this should be investigated.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I131493abd90283e9ca897f904e00c25d26e3d8d3
Reviewed-on: https://review.whamcloud.com/47126
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15727 lod: honor append_pool with default composite layouts 14/47014/3
John L. Hammond [Thu, 7 Apr 2022 16:31:07 +0000 (11:31 -0500)]
LU-15727 lod: honor append_pool with default composite layouts

In lod_get_default_lov_striping(), correct the handling of composite
default layouts in the case where append_stripe_count is nonzero.
Align the names of the append members of struct dt_allocation_hint
with the mdd params. Remove the unused dah_mode member of struct
dt_allocation_hint.

Add sanity test_27U() to verify.

Fixes: e2ac6e1eaa ("LU-9341 lod: Add special O_APPEND striping")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I66b426d24d6476fb483397f290229983f3da4be5
Reviewed-on: https://review.whamcloud.com/47014
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15922 sec: new connect flag for name encryption 74/47574/5
Sebastien Buisson [Thu, 9 Jun 2022 08:00:52 +0000 (10:00 +0200)]
LU-15922 sec: new connect flag for name encryption

Introduce OBD_CONNECT2_ENCRYPT_NAME connection flag for compatibility
with older versions that do not support name encryption.
When server side does not have this flag, client side is forced to
null encryption for file names. And client needs to use old xattr to
store encryption context.

Also update tests in sanity-sec to exercise name encryption only if
server side supports it.

Fixes: ed4a625d88 ("LU-13717 sec: filename encryption - digest support")
Test-Parameters: serverversion=2.14 testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 fstype=ldiskfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I446a4caba8e45821d701628a14c96f03cb6c4525
Reviewed-on: https://review.whamcloud.com/47574
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15952 doc: improvement on lfs-project doc 41/47641/2
Lei Feng [Thu, 16 Jun 2022 02:45:51 +0000 (10:45 +0800)]
LU-15952 doc: improvement on lfs-project doc

Describe 'lfs project -C' clearly.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: I1e0f70a9116265edac993b12b775bd57c8587d40
Reviewed-on: https://review.whamcloud.com/47641
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15942 utils: ofd_access_log_reader exit status 25/47625/2
John L. Hammond [Tue, 14 Jun 2022 13:32:54 +0000 (08:32 -0500)]
LU-15942 utils: ofd_access_log_reader exit status

If no OSTs are mounted then the ofd module may not be leaded and hence
/dev/lustre-access-log/control may not exist. In
ofd_access_log_reader, if --exit-on-close is used then we should
handle this consistently with the case of no access logs by exiting
with status 0.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I91b059bee8941501f2d207d2a48d1ea5ad40ae99
Reviewed-on: https://review.whamcloud.com/47625
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15931 tests: Escape * in log() 97/47597/3
Oleg Drokin [Sat, 11 Jun 2022 04:28:19 +0000 (00:28 -0400)]
LU-15931 tests: Escape * in log()

So it does not print every file name in test description in dmesg

Test-Parameters: trivial testlist=sanity env=ONLY=160s
Fixes: f60b307c50 ("LU-14699 mdd: proactive changelog garbage collection")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Change-Id: I300fde0f71ef15a5c6573a67324944ba8d53f8e3
Reviewed-on: https://review.whamcloud.com/47597
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-8621 utils: cmd help to stdout or short cmd error 62/47162/3
Aleksei Alyaev [Thu, 23 Dec 2021 08:48:22 +0000 (11:48 +0300)]
LU-8621 utils: cmd help to stdout or short cmd error

- Changed to print command help to stdout
- Changed to output short error message for an unrecognized command

Test-Parameters: trivial
Signed-off-by: Aleksei Alyaev <aalyaev@ddn.com>
Change-Id: I67616ddb576e3347a2da130b3a731a6bf8730185
Reviewed-on: https://review.whamcloud.com/47162
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-14555 lnet: asym route inconsistency warning 18/46918/6
Gian-Carlo DeFazio [Wed, 23 Mar 2022 23:25:56 +0000 (16:25 -0700)]
LU-14555 lnet: asym route inconsistency warning

lnet_check_route_inconsistency() checks for inconsistency between
the lr_hops and lr_single_hop values of a route.

A warning is currently emitted if the route is not single hop
and the hop count is either 1 or LNET_UNDEFINED_HOPS.

To emit the warning, add the requirement that
avoid_asym_router_failure is enabled.

Test-Parameters: trivial
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Change-Id: Iaa26d25492e49b569ae5e81da9f00f162be3da59
Reviewed-on: https://review.whamcloud.com/46918
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-13335 ldiskfs: add projid to debug logs 69/46369/8
Andreas Dilger [Fri, 28 Jan 2022 05:14:49 +0000 (22:14 -0700)]
LU-13335 ldiskfs: add projid to debug logs

There is virtually no tracking of projid changes in osd-ldiskfs,
which makes it very difficult to debug operations therein.

Add some minimal debugging on the client and servers to log
the projid when it is changed, along with the affected FID.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibcf3f09ee243ebe052c8f9119383897072ce7057
Reviewed-on: https://review.whamcloud.com/46369
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15481 llog: Add LLOG_SKIP_PLAIN to skip llog plain 10/46310/5
Etienne AUJAMES [Tue, 25 Jan 2022 21:38:26 +0000 (22:38 +0100)]
LU-15481 llog: Add LLOG_SKIP_PLAIN to skip llog plain

Add the catalog callback return LLOG_SKIP_PLAIN to conditionally skip
an entire llog plain.

This could speedup the catalog processing for specific usages when a
record need to be access in the "middle" of the catalog. This could
be usefull for changelog with several users or HSM.

This patch modify chlg_read_cat_process_cb() to use LLOG_SKIP_PLAIN.
The main idea came from: d813c75d ("LU-14688 mdt: changelog purge
deletes plain llog")

**Performance test:**

* Environement:
2474195 changelogs record store on the mds0 (40 llog plain):
mds# lctl get_param -n mdd.lustrefs-MDT0000.changelog_users
current index: 2474195
ID    index (idle seconds)
cl1   0 (3509)

* Test
Access to records at the end of the catalog (offset: 2474194):
client# time lfs changelog lustrefs-MDT0000 2474194 >/dev/null

* Results
- with the patch:  real    0m0.592s
- without the patch: real    0m17.835s (x30)

Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr>
Change-Id: I887d5bef1f3a6a31c46bc58959e0f508266c53d2
Reviewed-on: https://review.whamcloud.com/46310
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15451 sec: read-only nodemap flag 49/46149/5
Sebastien Buisson [Fri, 14 Jan 2022 09:16:31 +0000 (10:16 +0100)]
LU-15451 sec: read-only nodemap flag

Add a new 'readonly_mount' property to nodemaps. When set, we return
-EROFS from server side if the client is not mounting read-only.
So the client will have to specify the read-only mount option to be
allowed to mount.

Fixes: 928714dddabb ("LU-5092 nodemap: save id maps to targets in new index file")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9931844ae46dfd5d724f592f8dfacc4a8011c7e3
Reviewed-on: https://review.whamcloud.com/46149
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15399 llite: dont restart directIO with IOCB_NOWAIT 47/46147/4
Qian Yingjin [Mon, 27 Dec 2021 03:23:45 +0000 (11:23 +0800)]
LU-15399 llite: dont restart directIO with IOCB_NOWAIT

It should hanlde FLR mirror retry and io_uring with IOCB_NOWAIT
flag differently.

int cl_io_loop(const struct lu_env *env, struct cl_io *io)
{
...
if (result == -EAGAIN && io->ci_ndelay) {
io->ci_need_restart = 1;
result = 0;
}
...
}

ssize_t
generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
{
...
if (iocb->ki_flags & IOCB_NOWAIT) {
if (filemap_range_has_page(mapping, iocb->ki_pos,
   iocb->ki_pos +
   count - 1))
return -EAGAIN;
...
}

In current code, it will restart I/O engine for read when get
-EAGAIN code.
However, for io_uring direct IO with IOCB_NOWAIT, if found that
there are cache pages in the current I/O range, it should return
-EAGAIN to the upper layer immediately. Otherwise, it will stuck
in an endless loop.

This patch also adds a tool "io_uring_probe" to check whether
the kernel supports io_uring fully.
The reason adding this check is because the rhel8.5 kernel has
backported io_uring:
cat /proc/kallsyms |grep io_uring
ffffffffa8510e10 W __x64_sys_io_uring_enter
ffffffffa8510e10 W __x64_sys_io_uring_register
ffffffffa8510e10 W __x64_sys_io_uring_setup
but the io_uring syscalls return -ENOSYS.

Test-Parameters: clientdistro=ubuntu2004 testlist=sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Id4374382e56e90d02349676891aa57b216b3deff
Reviewed-on: https://review.whamcloud.com/46147
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-7668 utils: add lctl del_ost 49/41449/12
Stephane Thiell [Tue, 9 Feb 2021 06:47:31 +0000 (22:47 -0800)]
LU-7668 utils: add lctl del_ost

Add helper command:

   lctl del_ost [--dryrun] --target fsname-OSTxxxx

Permanently disable an OST by altering the client and MDT llog
catalogs on MGS. The command finds all catalog records related to the
specified OST and cancel them. A --dryrun option is provided so that
the system administrator can see which records would have been
cancelled, but without actually cancelling them.

Signed-off-by: Stephane Thiell <sthiell@stanford.edu>
Change-Id: I58c4f10fa0f7164a40231e807698eb224cccf062
Reviewed-on: https://review.whamcloud.com/41449
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-13189 osd-zfs: add project id for old objects without ZFS_PROJID 09/47709/3
Li Dongyang [Thu, 23 Jun 2022 06:36:19 +0000 (16:36 +1000)]
LU-13189 osd-zfs: add project id for old objects without ZFS_PROJID

After project quota zpool upgrade, the ZFS_PROJID
flag could still be missing on some old objects.

We used to check for this and return ENXIO in
osd_declare_attr_set(), however the check is changed
by "LU-12309 osd-zfs: Support disabled project quotas".

Now if the target project id is the default
project id 0, we will pass the check in
osd_declare_attr_set() and trigger the assert in
osd_attr_set() later.

Instead of returning ENXIO, we could adjust the
attribute layout of the old objects to accommodate
project id.

Also add back the logic from "LU-14740 quota: reject
invalid project id on server side", which got removed
by "LU-14927 quota: move qsd_transfer to lquota module",
due to using GPL symbols.

Change-Id: Ib62fdd2a0e07f15ae12daf564273a249a54dd8ea
Fixes: 291e7196d3 ("LU-12309 osd-zfs: Support disabled project quotas")
Fixes: d2e8208e22 ("LU-14927 quota: move qsd_transfer to lquota module")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/47709
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15926 nrs: fix tbf realtime rules 85/47585/3
Etienne AUJAMES [Thu, 9 Jun 2022 20:50:06 +0000 (22:50 +0200)]
LU-15926 nrs: fix tbf realtime rules

tc_nsecs_resid should be reset to 0 when changing a rule otherwise
this could lead to mds crashes for realtime policies.

nrs_tbf_req_get(): ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs )

Fixes: d11fa2c27959 ("LU-9228 nrs: TBF realtime policies under congestion")
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I280acb42e104088c6b8750a0bb7bf9c50cf96e73
Reviewed-on: https://review.whamcloud.com/47585
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-6142 lnet: use list_first_entry() in lnet/lnet subdirectory. 88/47488/3
Mr NeilBrown [Tue, 31 May 2022 00:43:12 +0000 (20:43 -0400)]
LU-6142 lnet: use list_first_entry() in lnet/lnet subdirectory.

Convert
  list_entry(foo->next .....)
to
  list_first_entry(foo, ....)

in 'lnet/lnet'

In several cases the call is combined with
a list_empty() test and list_first_entry_or_null() is used

Test-Parameters: trivial testlist=sanity-lnet
Change-Id: I45e1bdfe41854c88af98ebf24797f72a68b11dc3
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/47488
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15713 lnet: Ensure round robin across nets 76/46976/7
Chris Horn [Thu, 31 Mar 2022 01:49:46 +0000 (20:49 -0500)]
LU-15713 lnet: Ensure round robin across nets

Introduce a global net sequence number and a peer sequence number.
These sequence numbers are used to ensure round robin selection of
local NIs and peer NIs across nets.

Also consolidate the sequence number accounting under
lnet_handle_send(). Previously the sequence number increment for
the final destination peer net/peer NI on a routed send was done
in lnet_handle_find_routed_path().

Some cleanup that is also in this patch:
 - Redundant check of null src_nid is removed from
   lnet_handle_find_routed_path() (LNET_NID_IS_ANY handles null arg)
 - Avoid comparing best_lpn with itself in
   lnet_handle_find_routed_path() on the first loop iteration
 - In lnet_find_best_ni_on_local_net() check whether we have
   a specified lp_disc_net_id outside of the loop to avoid doing
   that work on each loop iteration.

Added some debug statements to print information used when selecting
peer net/local net.

HPE-bug-id: LUS-10871
Test-Parameters: trivial
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ide07e832deda85735042835e3097b9bf92e1e4b0
Reviewed-on: https://review.whamcloud.com/46976
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-10391 lnet: change lnet_*_peer_ni to take struct lnet_nid 25/44625/6
Mr NeilBrown [Tue, 18 Jan 2022 17:14:59 +0000 (12:14 -0500)]
LU-10391 lnet: change lnet_*_peer_ni to take struct lnet_nid

lnet_add_peer_ni() and lnet_del_peer_ni() now take
struct lnet_nid rather than lnet_nid_t.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I48bbed96fddea509b27dee131e134aa7b35ae68c
Reviewed-on: https://review.whamcloud.com/44625
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-10391 lnet: discard some peer_ni lookup functions 24/44624/7
Mr NeilBrown [Thu, 16 Jun 2022 18:32:21 +0000 (14:32 -0400)]
LU-10391 lnet: discard some peer_ni lookup functions

lnet_nid2peerni_locked(), lnet_peer_get_ni_locked(),
lnet_find_peer4(), and lnet_find_peer_ni_locked() each have few users
left and that can call be change to use alternate versions which take
'struct lnet_nid' rather than 'lnet_nid_t'.

So convert all those callers over, and discard the older functions.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9f0ebd0631c2e4160c3198aa37f16b45027bce3d
Reviewed-on: https://review.whamcloud.com/44624
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-6142 obdclass: checkpatch cleanup of obd_mount_server.c 59/47259/6
James Simmons [Mon, 6 Jun 2022 14:30:05 +0000 (10:30 -0400)]
LU-6142 obdclass: checkpatch cleanup of obd_mount_server.c

Address the many issues reported by checkpatch.pl. Replace __u**
with u** since it is kernel only code. Move lustre_tgt
registeration from obd_mount.c to obd_mount_server.c.

Change-Id: If06d120434ecb200f7a265167a537b2c98519670
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47259
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-11695 som: disabling xattr cache for LSOM on client 11/33711/3
Qian Yingjin [Fri, 23 Nov 2018 08:10:54 +0000 (16:10 +0800)]
LU-11695 som: disabling xattr cache for LSOM on client

To obtain uptodate LSOM data, currently a client needs to set
llite.*.xattr_cache =0 to disable the xattr cache on client
completely. This leads that other kinds of xattr can not be cached
on the client too.
This patch introduces a heavy-weight solution to disable caching
only for LSOM xattr data ("trusted.som") on client.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iab5ef3030b05ac09184d01f2a3a8ed92ff1cf26b
Reviewed-on: https://review.whamcloud.com/33711
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-14642 tests: skip sanity-flr/100 for old servers 67/47567/2
Andreas Dilger [Wed, 8 Jun 2022 14:40:20 +0000 (08:40 -0600)]
LU-14642 tests: skip sanity-flr/100 for old servers

The new FLR FSX mode in sanity-flr test_100 triggers LU-13730 when
run on old servers.  Skip it in this case.

Test-Parameters: trivial testlist=sanity-flr env=ONLY=100
Test-Parameters: serverversion=2.14 testlist=sanity-flr env=ONLY=100
Fixes: 90ba8b4ac360 ("LU-14642 test: add fsx mirror file test mode")
Fixes: 571f3cf11159 ("LU-13730 lod: don't confuse stale with primary flag")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I595bba6d5ab318ae591d80704c70998af9166de1
Reviewed-on: https://review.whamcloud.com/47567
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15282 tests: improve sanity test_51d coverage 54/46154/11
Andreas Dilger [Mon, 17 Jan 2022 23:24:48 +0000 (16:24 -0700)]
LU-15282 tests: improve sanity test_51d coverage

Improve sanity test_51d to test all different stripe counts, rather
than only striping over all OSTs.  With the current default test
config there are 7 OSTs, and this does not cover some test cases.

Test-Parameters: trivial testlist=sanity env=ONLY=51d ostcount=3
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=4
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=5
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=6
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=7
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=8
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=9
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=10
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=11
Test-Parameters: testlist=sanity env=ONLY=51d ostcount=12
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icc694d037e53f7bf966aff8ca1070d42ac3ebbe5
Reviewed-on: https://review.whamcloud.com/46154
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15720 dne: add crush2 hash type 15/47015/8
Andreas Dilger [Tue, 12 Apr 2022 23:18:10 +0000 (17:18 -0600)]
LU-15720 dne: add crush2 hash type

The original "crush" hash type has a significant error with files
that have all-number suffixes, or suffixes that have non-alpha
characters in them.  These files will all be placed on the same
MDT as the base filename, which causes MDT imbalance.

Add a "crush2" hash type that has more stringent checks for the
suffix, so that it doesn't consider all-digit suffixes, or files
that only have a '.' at the right offset, as temporary files.

Test that the "broken" all-digit or extra-'.' filenames are hashed
properly with "crush2".  We also need to confirm that the old "crush"
hash has not changed (for name lookup compatibility) and still has
the original "bad hashing" bug that puts all files on the same MDT.

Fix handling of types beyond MDT_HASH_TYPE_CRUSH when creating dirs.

Fix debug layout printing of hash_type in more parts of the code.
Don't flood console if hash type is unrecognized in the future.

Fixes: 0a1cf8da8069 ("LU-11025 dne: introduce new directory hash type 'crush'")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1ce34b8f3af44432f55307ebc6906677c6179d1d
Reviewed-on: https://review.whamcloud.com/47015
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15956 gss: allow build without ssk 81/47681/6
Sebastien Buisson [Wed, 6 Jul 2022 04:22:20 +0000 (21:22 -0700)]
LU-15956 gss: allow build without ssk

The GSS part of Lustre should be able to build without SSK, in case
some SSK requirements are not met at configure time.

Test-Parameters: trivial
Test-Parameters: testgroup=review-dne-selinux-ssk-part-1
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ieacbcb5db77fcc12cc13579785e640857ce7fb02
Reviewed-on: https://review.whamcloud.com/47681
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Stephane Thiell <sthiell@stanford.edu>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15973 build: remove AC_DEFINE(__state, state, ...) 99/47799/2
Jian Yu [Mon, 27 Jun 2022 17:30:16 +0000 (10:30 -0700)]
LU-15973 build: remove AC_DEFINE(__state, state, ...)

RHEL 8.6 build failed with MLNX_OFED 5.6-2.0.9.0 as follows:

error: 'struct task_struct' has no member named 'state'; did you mean '__state'?
 #define __state state
                 ^~~~~

The failure was introduced by
commit bb7c82f13e7a01891edcdf0626c6fb91a240e56e and a proper
way to resolve the original issue was in
commit c04adbcd76725a360f411f09c63df785bf7db426. So, let's
remove the improper way AC_DEFINE(__state, state, ...).

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Change-Id: Icbc897cf5870352311262af8bd59ec24ea9d7301
Fixes: bb7c82f13e7 ("LU-15795 kernel: RHEL 8.6 server support")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47799
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15962 build: add in-kernel Module.symvers to symbol path 99/47699/2
Jian Yu [Wed, 22 Jun 2022 18:52:45 +0000 (11:52 -0700)]
LU-15962 build: add in-kernel Module.symvers to symbol path

After building Lustre with in-kernel OFED, installing
ko2iblnd module hit the following errors:

ko2iblnd: disagrees about version of symbol __ib_alloc_pd
ko2iblnd: Unknown symbol __ib_alloc_pd (err -22)
ko2iblnd: disagrees about version of symbol rdma_resolve_addr
ko2iblnd: Unknown symbol rdma_resolve_addr (err -22)

Those exported symbols are contained in in-kernel Module.symvers,
which should be added to the symbol path KBUILD_EXTRA_SYMBOLS.

Test-Parameters: trivial

Change-Id: Ic30caa7079af00a452ea24e7e982a856874af702
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47699
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15940 build: add a required dependency for libmount 19/47619/5
Jian Yu [Sun, 19 Jun 2022 07:47:01 +0000 (00:47 -0700)]
LU-15940 build: add a required dependency for libmount

The Lustre client utilities (mount/umount) have
an optional dependency on libmount to update utab.
However, libmount has been introduced in util-linux
since 2.18 in 2010. There is no need to make the
dependency as optional.

Test-Parameters: trivial clientdistro=ubuntu2004
Test-Parameters: trivial clientdistro=el8.5

Change-Id: I4b965a5ce6cb6fc5d2061a53c44ef9b709ebab49
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47619
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15652 build: On Debian detect -common kernel headers 36/46836/6
Shaun Tancheff [Wed, 25 May 2022 16:46:50 +0000 (23:46 +0700)]
LU-15652 build: On Debian detect -common kernel headers

Check for a matching /usr/src/linux-headers-<ver>-common/
and update the --with-linux argument accordingly.

Also move LC_GLIBC_SUPPORT_COPY_FILE_RANGE outside
of utils as this also breaks the dkms build on Debian
with 'static' follows 'non static' declairation of
copy_file_range.

Fixes: e6d1968fbbad ("LU-13903 build: Move GLIBC/openssl checks to where needed")
HPE-bug-id: LUS-10826
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I6e4f0b27eba6c5b07cda14f064e57aa9c93ae3cc
Reviewed-on: https://review.whamcloud.com/46836
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15703 ldiskfs: Disable unused fast commit buffer 43/46943/8
Shaun Tancheff [Tue, 31 May 2022 00:27:52 +0000 (20:27 -0400)]
LU-15703 ldiskfs: Disable unused fast commit buffer

Linux commit v5.9-rc7-39-g6866d7b3f2bb
    ext4 / jbd2: add fast commit initialization

Disable journal fast commit buffer via a mount option because it is
not used by lustre's ldiskfs since it will break recovery.

Linux commit v5.10-rc2-9-gede7dc7fa0af
    jbd2: rename j_maxlen to j_total_len and add
          jbd2_journal_max_txn_bufs

Change osd_transaction_size to use jbd2_journal_get_max_txn_bufs
and provide a jbd2_journal_get_max_txn_bufs when it is not
provided.

Test-Parameters: trivial
HPE-bug-id: LUS-10858
Fixes: c93a3e5b15 ("LU-14195 ldiskfs: update patches for Linux 5.10")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I9bffc3559a8bbce9d4c1c2b6692cb8518f3f991a
Reviewed-on: https://review.whamcloud.com/46943
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
4 months agoLU-13562 build: get correct kernel flavor for SLES 94/47594/2
Jian Yu [Sat, 11 Jun 2022 03:57:54 +0000 (20:57 -0700)]
LU-13562 build: get correct kernel flavor for SLES

This patch fixes lustre.spec.in to get correct kernel flavor
for SLES when kobjdir is detected as /lib/modules/%{_kver}/build.

Test-Parameters: trivial clientdistro=sles15sp3

Change-Id: I350032af383ea8b7f48accd93e5cd11c571e6620
Fixes: d746e64fe1 ("LU-13562 build: SUSE build support for azure, cray_ari_s")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15838 autoconf: fix use of obsolete macros 88/47288/3
Jian Yu [Mon, 30 May 2022 21:02:50 +0000 (14:02 -0700)]
LU-15838 autoconf: fix use of obsolete macros

This patch fixes the following warnings when using autoconf 2.71:

configure.ac:2: warning: AC_INIT: not a literal:
                "m4_esyscmd(sh -c "./LUSTRE-VERSION-GEN | tr -d '\n'")"
configure.ac:10: warning: The macro `AC_CANONICAL_SYSTEM' is obsolete.
configure.ac:16: warning: The macro `AC_PROG_LIBTOOL' is obsolete.
configure.ac:24: warning: The macro `AC_HELP_STRING' is obsolete.

Like m4_esyscmd, macro m4_esyscmd_s (introduced in autoconf 2.64)
expands to the result of running command in a shell. The difference
is that any trailing newlines are removed.

Since autoconf 2.50, macro 'AC_CANONICAL_TARGET' has been the new name
of 'AC_CANONICAL_SYSTEM':
AU_ALIAS([AC_CANONICAL_SYSTEM], [AC_CANONICAL_TARGET])

Since autoconf 2.58, macro 'AS_HELP_STRING' has been added to replace
'AC_HELP_STRING'.

Since libtool 2.0, new 'LT_INIT' interface has been added to replace
'AC_PROG_LIBTOOL'.

Change-Id: I3c06c21460d7a2cf643fe825e72a26a5416609cf
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47288
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15968 build: update libssl3 51/47751/2
Minh Diep [Fri, 24 Jun 2022 06:19:03 +0000 (23:19 -0700)]
LU-15968 build: update libssl3

In Ubuntu 22.04 libssl1.1 has been superceded by libssl3

Test-Parameters: trivial clientdistro=ubuntu2004

Change-Id: Ic2504c3a40d5c756c2da7b81edb1501fb6b44712
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15005 build: wrong dependencies for lustre-client-modules deb package 47/47747/2
Alex Deiter [Fri, 24 Jun 2022 06:04:55 +0000 (23:04 -0700)]
LU-15005 build: wrong dependencies for lustre-client-modules deb package

Fixed dependencies for DKMS deb package:
- added autocon, automake and libtool
- added bison and flex
- added required dev packages
- added linux-base and linux-image
- added python3-distutils-extra to fix build on Ubuntu 16.04

Change-Id: Ic1d05155cd8ad056dece1d22d0f040695d038652
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/47747
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15967 build: configure script does not check for required build tools 44/47744/2
Alex Deiter [Fri, 24 Jun 2022 05:55:58 +0000 (22:55 -0700)]
LU-15967 build: configure script does not check for required build tools

- added check for flex and bison
- added requirement for build kernel modules

Change-Id: I4f4f19ea44f3cd8f69482d950970bf701e81f7ec
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/47744
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15939 build: configure script changes system header and config files 04/47604/2
Alex Deiter [Sun, 12 Jun 2022 19:26:26 +0000 (12:26 -0700)]
LU-15939 build: configure script changes system header and config files

Remove the SUBDIRS target from configure tests.
Since Linux kernel version 2.6.x we can use target M.

Test-Parameters: trivial clientdistro=ubuntu2004
Test-Parameters: trivial clientdistro=el8.5

Change-Id: I8e59bdaf2d0e4e08a659e08f63a14472fba72eb2
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/47604
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
4 months agoLU-15914 lnet: Fix null md deref for finalized message 46/47546/4
Chris Horn [Mon, 6 Jun 2022 18:09:03 +0000 (13:09 -0500)]
LU-15914 lnet: Fix null md deref for finalized message

When a message is finalized the lnet_msg:msg_md field may be cleared
(see lnet_finalize() -> lnet_msg_detach_md()).
When an LNet router is forwarding such message, or if an ACK has been
requested for such a message, then the NULL msg_md may be deref'd in
lnet_get_best_ni(). Check for this in lnet_get_best_ni() before
dereferencing the MD.

It may also be dereferenced in kiblnd_send(), so check for this
situation there, too. Some style cleanup is included in
kiblnd_send().

Test-Parameters: trivial
Fixes: 959304eac7 ("LU-15189 lnet: fix memory mapping.")
HPE-bug-id: LUS-10997
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I3cfdc8d342bd3b49a61d1ce6c31a245848accf8f
Reviewed-on: https://review.whamcloud.com/47546
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15910 tests: skip sanity/413g for SSK 00/47800/3
Andreas Dilger [Mon, 27 Jun 2022 18:24:37 +0000 (12:24 -0600)]
LU-15910 tests: skip sanity/413g for SSK

When running sanity test_413g under review-dne-selinux-ssk-part-1
it intermittently fails.  Temporarily disable this subtest for
this config until the problem is understood and fixed.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iacda6add8baabd479ea05dff06867f1a7afb23ce
Reviewed-on: https://review.whamcloud.com/47800
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15910 llite: enforce ROOT default on subdir mount 18/47518/4
Lai Siyao [Sat, 21 May 2022 02:21:38 +0000 (22:21 -0400)]
LU-15910 llite: enforce ROOT default on subdir mount

In subdirectory mount, the filesystem-wide default LMV doesn't take
effect. This fix includes the following changes:
* enforce the filesystem-wide default LMV on subdirectory mount if
  it's not set separately.
* "lfs getdirstripe -D <subdir_mount>" should print the
  filesystem-wide default LMV.

Add sanity test_413g.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I26de9d02872f0df8918b4ef0765b6b18b84794e6
Reviewed-on: https://review.whamcloud.com/47518
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15900 hsm: don't return error on state change during mount 73/47473/2
Mikhail Pershin [Sat, 28 May 2022 06:59:03 +0000 (09:59 +0300)]
LU-15900 hsm: don't return error on state change during mount

HSM coordinator is started in stopped state always, but
mount may have hsm_control parameters 'disabled'. Such
parameter cause wrong state change so mount would fail
with error.

Treat parameter change from 'stopping/stopped' to 'disabled'
as not critical error, keep state unchanged and report
no error back to a caller.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I9d1366423391971b9511c46b6aed39d21ebf637c
Reviewed-on: https://review.whamcloud.com/47473
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15860 socklnd: Duplicate ksock_conn_cb 61/47361/3
Chris Horn [Thu, 12 May 2022 18:16:10 +0000 (13:16 -0500)]
LU-15860 socklnd: Duplicate ksock_conn_cb

If two threads enter ksocknal_add_peer(), the first one to acquire
the ksnd_global_lock will create a ksock_peer_ni and associate a
ksock_conn_cb with it.

When the second thread acquires the ksnd_global_lock it will find the
existing ksock_peer_ni, but it does not check for an existing
ksock_conn_cb. As a result, it overwrites the existing ksock_conn_cb
(ksock_peer_ni::ksnp_conn_cb) and the ksock_conn_cb from the first
thread becomes stranded.

Modify ksocknal_add_peer() to check whether the peer_ni has an
existing ksock_conn_cb associated with it

Fixes: 7766f01e89 ("LU-13641 socklnd: replace route construct")
HPE-bug-id: LUS-10956
Test-Parameters: trivial
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I6c0190a0c1d3321ddd85c763b86ad1f0d32cf2b9
Reviewed-on: https://review.whamcloud.com/47361
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15855 enc: enc-unaware clients get ENOKEY if file not found 49/47349/2
Sebastien Buisson [Fri, 13 May 2022 15:03:22 +0000 (17:03 +0200)]
LU-15855 enc: enc-unaware clients get ENOKEY if file not found

To reduce issues with applications running on clients without keys
or without fscrypt support that check for the existence of a file in
an encrypted directory, return -ENOKEY instead of -ENOENT.
For encryption-unaware clients, this is done on server side in the
mdt layer, by checking if clients have the OBD_CONNECT2_ENCRYPT
connection flag.
For clients without the key, this is done in llite when the searched
filename is not in encoded form.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9a3b7af3a856b7fc7222c61a308ad23168869d57
Reviewed-on: https://review.whamcloud.com/47349
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15839 tests: correct the ZFS grace time for sanity-quota 4a 89/47289/3
Etienne AUJAMES [Wed, 11 May 2022 07:26:01 +0000 (09:26 +0200)]
LU-15839 tests: correct the ZFS grace time for sanity-quota 4a

For  sanity-quota 4a, the grace time is increased from 12s to 20s but
not actually set on filesystem.

Fixes: 3e4c3fdc ("LU-6836 test: re-add test 4a to sanity-quota for ZFS")
Test-Parameters: fstype=zfs testlist=sanity-quota env=ONLY=4a,ONLY_REPEAT=100
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I2324e818a42a19bc9928f127b1622f1e5274db1f
Reviewed-on: https://review.whamcloud.com/47289
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
5 months agoLU-15420 build: fixes to support building on Ubuntu 22.04 LTS 33/47133/5
James Simmons [Mon, 25 Apr 2022 17:42:32 +0000 (13:42 -0400)]
LU-15420 build: fixes to support building on Ubuntu 22.04 LTS

Lustre uses the glibc stdarg.h instead of the kernel's version which
causes the following build issue.

lustre/include/lu_object.h:35,
/usr/lib/gcc/x86_64-linux-gnu/11/include/stdarg.h:52: note: this is the
location of the previous definition
        #define va_copy(d,s)    __builtin_va_copy(d,s)

The solution is to use the kernels version of stdarg.h

The second build issue :
update_trans.c:1608:30: error: ‘struct task_struct’ has no member named
                        ‘state’; did you mean ‘__state’?

is due Linux commit 2f064a59a11ff9bc22e52e9678bc601404c7cb34
(sched: Change task_struct::state). The state field was
changed and the barrier macros READ_ONCE()/WRITE_ONCE()
are used to access it now which is the proper thing to do.
Since the check in update_trans.c is equivalent to testing
if the kernel thread is not running, since TASK_RUNNING == 0,
we can just change the code to use task_is_running(). The
task_is_running() was introduced in 5.13.

Test-Parameters: trivial
Change-Id: Ib5985b187c3013fbc513e9962a5f27bed4996f5b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47133
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15055 lod: run qmt_pool_* only from the MDT0000 config 59/47059/6
Etienne AUJAMES [Wed, 13 Apr 2022 14:43:12 +0000 (16:43 +0200)]
LU-15055 lod: run qmt_pool_* only from the MDT0000 config

On the first mds (with MDT0000/QMT0000), if there is more than one MDT
target, qmt_pool_{new/del/rem/add} functions will be call several
times on QMT0000 for the same pool.

This resulting to the following error in dmseg:
LustreError: 5659:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't
scratch-QMT0000 scratch-OST0000_UUID pool pool1: rc = -17

This patch run qmt_pool_* only from a record config from the MDT0000.
The qmt_pool_add_rem() dmesg error is checked on sanity-quota test_1b.

Test-Parameters: mdtcount=2 mdscount=1 testlist=sanity-quota
Fixes: 09f9fb32 ("LU-11023 quota: quota pools for OSTs")
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: Ia6b712abe25a4d68770753e3408c3321181db1aa
Reviewed-on: https://review.whamcloud.com/47059
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15117 ofd: don't take lock for dt_bufs_get() 29/47029/5
Alex Zhuravlev [Mon, 11 Apr 2022 08:30:44 +0000 (11:30 +0300)]
LU-15117 ofd: don't take lock for dt_bufs_get()

osd_bufs_get() allocates the pages and can cause new transactions
as part of memory release procedure. this would break Lustre's
"start a transaction, then do locking" rule.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I58a52af8e2fbbc4823aafc133893e1defedf99b7
Reviewed-on: https://review.whamcloud.com/47029
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15625 sec: fix set_param -P for root squash 39/46739/2
Sebastien Buisson [Tue, 8 Mar 2022 11:07:47 +0000 (12:07 +0100)]
LU-15625 sec: fix set_param -P for root squash

lctl set_param -P needs to be fixed for root squash in order
to properly propagate to clients.
On server (MDT) side, the paramaters are mdt.*.*squash*.
On client side, the parameters are llite.*.*squash*.
So if the setting on MGS side is done as
set_param -P *.*.root_squash and
*.*.nosquash_nids, it works fine. But if it is done as
set_param -P mdt.*.root_squash, this setting is not applied to
client side.
The solution is to have the settings translated into
*.*.root_squash and *.*.nosquash_nids to make both sides happy.

Also, to ensure client vs. server consistency, prevent
set_param -P llite.*.root_squash.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I8984c9d9024d7a053e548ed15d1321b281d9940f
Reviewed-on: https://review.whamcloud.com/46739
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15706 llog: deal with "SKIP" pool llog records correctly 51/46951/9
Emoly Liu [Mon, 25 Apr 2022 02:56:55 +0000 (10:56 +0800)]
LU-15706 llog: deal with "SKIP" pool llog records correctly

If the requested "start" llog records by one ioctl are just the
useful ones "protected" by one marker, they will don't know their
cfg_flags(e.g. "SKIP" or not), and then will be mis-labeled in
class_config_yaml_output().
To fix this issue, this patch does the following changes:
- In kernel space, remember the marker cfg_flags for 10 records
  earlier than the requested "start" in function llog_print_cb(),
  so that the "start" can be output with its correct cfg_flags.
- In user space, since the pool_new/add record will be marked as
  "SKIP" if its corresponding pool_destroy/remove record is logged
  later in mgs_pool_cmd(), these "SKIP" records won't be printed
  by function jt_llog_print_iter(), so lpd_ost_num doesn't need to
  be decreased in callback function llog_search_pool_cb() as well,
  otherwise, it will cause "lctl pool_destroy" error.

Test-Parameters: standalonemgs=true testlist=ost-pools
Test-Parameters: standalonemgs=true testlist=conf-sanity env=ONLY=123,HONOR_EXCEPT=y
Fixes: 2a5b50d20717 ("LU-15142 lctl: fixes for set_param -P and llog_print")
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: Ie45127ac8b80a75eaeb7158559c690da52eef103
Reviewed-on: https://review.whamcloud.com/46951
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-10994 clio: remove cpo_assume, cpo_unassume, cpo_fini 73/47373/4
John L. Hammond [Mon, 16 May 2022 21:42:44 +0000 (16:42 -0500)]
LU-10994 clio: remove cpo_assume, cpo_unassume, cpo_fini

Remove the cl_page methods cpo_assume, cpo_unassume, and
cpo_fini. These methods were only implemented by the vvp layer and so
they can be easily inlined into cl_page_assume() and
cl_page_unassume(). Remove vvp_page_delete() by inlining its contents
to cl_page_delete0().

Change-Id: I260c5593983bac6742cf7577c26a4903e95ceb7c
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47373
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-10994 clio: remove cpo_own and cpo_disown 72/47372/4
John L. Hammond [Wed, 4 May 2022 20:44:03 +0000 (15:44 -0500)]
LU-10994 clio: remove cpo_own and cpo_disown

Remove the cpo_own and cpo_disown methods from struct
cl_page_operations. These methods were only implemented by the vvp
layer so they can be inlined into cl_page_own0() and
cl_page_disown(). Move most of vvp_page_discard() and all of
vvp_transient_page_discard() into cl_page_discard().

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I3f156d6ca3e4ea11c050b2addda38e84a84634b9
Reviewed-on: https://review.whamcloud.com/47372
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-10994 clio: remove cl_page_export() and cl_page_is_vmlocked() 41/47241/5
John L. Hammond [Wed, 4 May 2022 21:03:34 +0000 (16:03 -0500)]
LU-10994 clio: remove cl_page_export() and cl_page_is_vmlocked()

Remove cl_page_export() and cl_page_is_vmlocked(), replacing them with
direct calls to PageSetUptodate() and PageLoecked().

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I883d1664f4afc7a1d4006f9f4833db8125c0e8f5
Reviewed-on: https://review.whamcloud.com/47241
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-10994 echo: remove client operations from echo objects 40/47240/5
John L. Hammond [Fri, 6 May 2022 17:30:10 +0000 (12:30 -0500)]
LU-10994 echo: remove client operations from echo objects

Remove the client (io, page, lock) operations from echo_client
objects. This will facilitate the simplification of CLIO.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: If9e55c7d54c171aa2e1bcf272641c2bd6be8ad48
Reviewed-on: https://review.whamcloud.com/47240
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-10994 test: remove netdisk from obdfilter-survey 39/47239/6
John L. Hammond [Fri, 6 May 2022 17:06:57 +0000 (12:06 -0500)]
LU-10994 test: remove netdisk from obdfilter-survey

Remove the netdisk case from obdfilter-survey. Remove subtests that
use echo_client over osc devices.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I260001241cee3027f68e62077e5817221bd0c08b
Reviewed-on: https://review.whamcloud.com/47239
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15593 mdt: Add option to disable use of SOM 83/46683/7
Patrick Farrell [Mon, 7 Mar 2022 20:31:34 +0000 (15:31 -0500)]
LU-15593 mdt: Add option to disable use of SOM

There's currently no way to disable use of strict SOM,
which is a problem if there's ever a SOM bug.  This is
tricky to do from the client, but easy on the MDT.

The test just verifies that the size stays the same with
strict SOM disabled, because there's no easy way to check
SOM is disabled unless SOM is broken.  (Since it gives the
same value for stat.)

Note this patch requires LU-15609 to be fixed on the client
or the client will see size as 0 when SOM is disabled.

Test-Parameters: testlist=sanity-flr env="ONLY=44e,ONLY_REPEAT=20"
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I338fa7e4dd423b07df0f3c5bad4ec6f02e935fea
Reviewed-on: https://review.whamcloud.com/46683
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15611 osd-zfs: Cleanup while mount failed 78/46678/4
Yang Sheng [Wed, 2 Mar 2022 13:25:59 +0000 (21:25 +0800)]
LU-15611 osd-zfs: Cleanup while mount failed

Need clean up in error out path in osd-zfs.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I47d9ee9483acb8e1d60c77e8cfc481902a1535ac
Reviewed-on: https://review.whamcloud.com/46678
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-14067 tests: re-enable gcc compile test for PPC + ARM 09/46409/3
James Simmons [Tue, 31 May 2022 00:06:15 +0000 (20:06 -0400)]
LU-14067 tests: re-enable gcc compile test for PPC + ARM

The development environment earlier was incorrectly setup in the
test bed. With the renewal of the ARM / PPC testbed the
development environment should work so re-enable the gcc test.

Test-Parameters: trivial clientdistro=el8.5 clientarch=aarch64 testlist=sanity env=ONLY=400
Test-Parameters: trivial clientdistro=el8.5 clientarch=ppc64le testlist=sanity env=ONLY=400
Test-Parameters: trivial clientdistro=el8.5 clientarch=aarch64 testlist=sanity-lnet env=ONLY=300
Test-Parameters: trivial clientdistro=el8.5 clientarch=ppc64le testlist=sanity-lnet env=ONLY=300

Change-Id: I95eb8e6f08402209aa350b81cd9517d0f8422632
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46409
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: xinliang <xinliang.liu@linaro.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15933 libcfs: fix configure check for iov_iter member 01/47601/4
Bobi Jam [Sat, 11 Jun 2022 17:18:18 +0000 (01:18 +0800)]
LU-15933 libcfs: fix configure check for iov_iter member

Kernel commit v5.13-rc1-12-g8cd54c1c8480 changed iov_iter member
->type to ->iter_type, add check at configure to cover this change.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I8da26e1f6c6242a3260b458770ec34a65aecaf7d
Reviewed-on: https://review.whamcloud.com/47601
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-12511 test: use proper UAPI header for fsx test app 65/47565/2
James Simmons [Wed, 8 Jun 2022 13:20:38 +0000 (09:20 -0400)]
LU-12511 test: use proper UAPI header for fsx test app

Using uapi as part of the path of the header forces the use of
the OpenSFS tree internal headers instead of an possible
external UAPI headers; which is the case for the native Linux
Lustre client. Just use linux/lustre path for UAPI headers
which works in all cases.

Change-Id: Ia46021f39a8f2e4ecae71846cbb76464617d12ab
Test-Parameters: trivial
Fixes: 90ba8b4ac ("LU-14642 test: add fsx mirror file test mode")
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47565
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15043 lod: check for spilling loops 83/45083/17
Alex Zhuravlev [Wed, 29 Sep 2021 06:16:01 +0000 (09:16 +0300)]
LU-15043 lod: check for spilling loops

at setting to avoid possible confusion.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I901b08f614c162607b1b5c6a992aa5b188fd8e75
Reviewed-on: https://review.whamcloud.com/45083
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15513 lod: iterate initialized stripe 28/47028/7
Bobi Jam [Mon, 11 Apr 2022 08:46:32 +0000 (16:46 +0800)]
LU-15513 lod: iterate initialized stripe

Fix an array out of bounds access.

Do not iterate over the component objects in lod_fill_mirrors() to
check non-rotational OSTs if the component is unintialized or it is
a MDT or RELEASED component.

Fixes: 591a990c61 ("LU-15513 lod: skip uninit component in lod_fill_mirrors")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I8530cd30fab750c47f15ee9278ec6d68871367cc
Reviewed-on: https://review.whamcloud.com/47028
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15609 flr: Don't assume RDONLY implies SOM 66/46666/5
Patrick Farrell [Mon, 7 Mar 2022 20:29:26 +0000 (15:29 -0500)]
LU-15609 flr: Don't assume RDONLY implies SOM

In lov_io_slice_mirror_init, the client code assumes that
the LCM_FL_RDONLY flag in the layout implies SOM and skips
glimpse if it sees one.  The RDONLY flag means the mirrors
are in sync, which has historically implied SOM is valid.

To start with, using LCM_FL_RDONLY to imply SOM is sort of
a layering violation.  SOM is only communicated from the
MDS when it is valid, and the client already skips glimpse
in that case, so this duplicates functionality from the
higher layers.

More seriously, patch:
"LU-14526 flr: mirror split downgrade SOM"
(https://review.whamcloud.com/43168/)
Made it possible to have LCM_FL_RDONLY but not strict SOM,
so this assumption is no longer correct.

The fix is to not look at LCM_FL_RDONLY when deciding
whether to glimpse a file for size.

Fixes: d87b6f4d1c ("LU-14526 flr: mirror split downgrade SOM")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5ed0a2124005bc58c1ed11681c9bd642cffcd1b5
Reviewed-on: https://review.whamcloud.com/46666
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15901 mdc: Remove entry from list before freeing 80/47480/3
Oleg Drokin [Mon, 30 May 2022 07:05:23 +0000 (03:05 -0400)]
LU-15901 mdc: Remove entry from list before freeing

mdc_changelog_cdev_init forgot to remove entries from list if
chardev allocation failed

Change-Id: Ic76b5320bf80c7f7f60c7682bda4bc37a0b300bd
Fixes: d0423abc1ad ("LU-12506 changelog: support large number of MDT")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47480
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
5 months agoLU-14535 quota: improve quota output format 99/43099/10
Andreas Dilger [Tue, 9 Mar 2021 05:59:42 +0000 (22:59 -0700)]
LU-14535 quota: improve quota output format

Make the output of the quota proc files more readable by removing
needless whitespace that causes it to wrap over a single line.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I80fbd42dea865dff1d106724dbf69946d23ebbe5
Reviewed-on: https://review.whamcloud.com/43099
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-6864 tests: properly skip sanity/245b in interop 56/47656/3
Andreas Dilger [Thu, 16 Jun 2022 19:25:50 +0000 (13:25 -0600)]
LU-6864 tests: properly skip sanity/245b in interop

Skip sanity test_245b if the server does not support multi_mod_rpcs.

Test-Parameters: trivial serverversion=2.14 testlist=sanity env=ONLY=245
Fixes: 23028efcae01 ("LU-6864 osp: manage number of modify RPCs in flight")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibd73625c64deaa99f6f2a515bc1214cbb83ebbe5
Reviewed-on: https://review.whamcloud.com/47656
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15795 kernel: RHEL 8.6 server support 92/47192/8
Jian Yu [Sun, 12 Jun 2022 07:37:53 +0000 (00:37 -0700)]
LU-15795 kernel: RHEL 8.6 server support

This patch makes changes to support RHEL 8.6 release
with kernel 4.18.0-372.9.1.el8 for Lustre server.

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Change-Id: I86407171b40d40a05379085444eee79e2ac272b3
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47192
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15795 kernel: new kernel [RHEL 8.6 4.18.0-372.9.1.el8] 80/47180/7
Jian Yu [Sun, 12 Jun 2022 07:14:14 +0000 (00:14 -0700)]
LU-15795 kernel: new kernel [RHEL 8.6 4.18.0-372.9.1.el8]

This patch makes changes to support new RHEL 8.6 release
for Lustre client.

Test-Parameters: trivial clientdistro=el8.6

Change-Id: Id738259ed94104c3a3c7bb5c1b853cfabad49405
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47180
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15918 build: update debhelper compatibility level to 11 50/47550/2
Jian Yu [Tue, 7 Jun 2022 01:38:51 +0000 (18:38 -0700)]
LU-15918 build: update debhelper compatibility level to 11

This patch updates debhelper compatibility level to 11 (which is
currently the recommended level) to resolve the following build
warnings while running "make debs":

Compatibility levels before 10 are deprecated (level 9 in use)

Test-Parameters: trivial clientdistro=ubuntu2004 \
testlist=sanity

Change-Id: Ie5c63f6a3eb29e9aeace1f008d8d492e7587adf2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47550
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Thomas Stibor <thomas@stibor.net>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15896 gss: ignore OpenSSLv3 deprecated API 85/47485/2
Sebastien Buisson [Mon, 30 May 2022 14:11:50 +0000 (16:11 +0200)]
LU-15896 gss: ignore OpenSSLv3 deprecated API

Lustre GSS code makes use of some OpenSSL API that has been
deprecated in v3, namely all the functions in the DH_* family.
Those would need to be replaced with their EVP_PKEY_* counterparts
but in the meantime we just use the OPENSSL_SUPPRESS_DEPRECATED
define to continue using the deprecated API.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I413a2af1299123b29ed2043188da0348ce9d0a5e
Reviewed-on: https://review.whamcloud.com/47485
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-14104 tests: sanity/123* shouldn't fail performance checks 12/40512/16
Alex Zhuravlev [Mon, 2 Nov 2020 07:13:44 +0000 (10:13 +0300)]
LU-14104 tests: sanity/123* shouldn't fail performance checks

running in VMs as CPU resource isn't strictly guaranteed usually.

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ieec4a89b921f7ccc198eb10513d4980ad3a20b51
Reviewed-on: https://review.whamcloud.com/40512
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
5 months agoLU-15908 build: fix debian rules 12/47512/2
Sebastien Buisson [Thu, 2 Jun 2022 10:05:02 +0000 (12:05 +0200)]
LU-15908 build: fix debian rules

A comment in debian/rules file breaks a commands-line continuation,
causing exported env vars to be dropped.

Fixes: a5084c2f2e ("LU-14937 build: re-use config cache in 'make rpms/debs'")
Test-Parameters: trivial
Signed-off-by: Ake Sandgren <ake.sandgren@hpc2n.umu.se>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0894199301c489b1961e043651c74d7c6c0089d2
Reviewed-on: https://review.whamcloud.com/47512
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15875 kernel: kernel update RHEL7.9 [3.10.0-1160.66.1.el7] 97/47397/2
Jian Yu [Thu, 19 May 2022 18:20:41 +0000 (11:20 -0700)]
LU-15875 kernel: kernel update RHEL7.9 [3.10.0-1160.66.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.66.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I9e8ab33edd6cacbbf895399962027827a1befd5b
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47397
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
5 months agoLU-15865 ptlrpc: don't report eviction for lwp 74/47374/2
Alex Zhuravlev [Tue, 17 May 2022 13:28:38 +0000 (16:28 +0300)]
LU-15865 ptlrpc: don't report eviction for lwp

lwp connections aren't persistent, so they found themselves
"evicted" after target's restart, which is not correct.
don't confuse people with false error.

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ia995eac2c4e1323690f42147aedc74f8748eacf0
Reviewed-on: https://review.whamcloud.com/47374
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>