LU-17522 build: Distribute clang build infrastructure Macro files: lustre-toolchain.m4 lustre-compiler-plugins.m4 and directory: cc-plugins Should be included in distributed files, unconditionally. Test-Parameters: trivial Fixes: d684885098 ("LU-16961 clang: plugins and build system integration") Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I6ddedd82c6180ffd1c4134fda6af6df6bd23dd34 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53991 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17226 build: create config option for l_getsepol Add a configuration option for l_getsepol. l_getsepol is build by default unless the --disable-l_getsepol option is given to configure. lustre.spec.in builds l_getsepol by default and has its dependencies as build requirements. The implicit configuration check for the dependency openssl-devel is removed and replaced by a BuildRequires. Test-Parameters: trivial Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov> Change-Id: If71a2a4a524047edbd2b31e6fac7a42f36a030bf Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52849 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17421 build: Update check for arc_prune_func_t parameters In OpenZFS 2.2.1 the code for arc_prune_async() was unified so that FreeBSD and Linux did not have their own implementation versions of the same code. Part of this update changed first parameter for the arc_prune_func_t to be an uint64_t. Without this patch, Lustre would not build with ZFS 2.2.1 because of a failure for incompatible pointer types for the arc_prunte_func_t function pointer passed to arc_add_prune_callback(). Test-Parameters: trivial Signed-off-by: Brian Atkinson <batkinson@lanl.gov> Change-Id: Iaa03cc9421f27a8517ce04817f04102de9adb86a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53664 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Akash B <akash-b@hpe.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
LU-17131 ldiskfs: Add Ubuntu 20.04.5 release 5.15 kernel Add support for Ubuntu 20.04.5 5.15 kernel similar to el9.2 with updated patches: ext4-corrupted-inode-block-bitmaps-handling-patches.patch ext4-data-in-dirent.patch ext4-dont-check-before-replay.patch ext4-inode-version.patch ext4-mballoc-extra-checks.patch ext4-prealloc.patch ext4-filename-encode.patch Tested with tag Ubuntu-hwe-5.15-5.15.0-91.101_20.04.1 Test-Parameters: trivial Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: Ic1b4b0f25a9ac984186cf4f37b5a73d93af93ebd Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52414 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Tested-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16791 utils: ZFS 2.2 const prop args ZFS 2.2 now expects const char * from certain interfaces in sys/nvpair.h. I updated the build system to detect if this is the case and if so update the paramters passed to certain functions in libmount_utils_zfs.c to account for these changes. Without this patch, Lustre master would not build with ZFS master and the 2.2 release candidates. Signed-off-by: Brian Atkinson <batkinson@lanl.gov> Change-Id: I0469eeff6dafa6c276fc616381530b6b679d9da1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52519 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Akash B <akash-b@hpe.com> Reviewed-by: Thomas Bertschinger <bertschinger@lanl.gov> Reviewed-by: Olaf Faaland <faaland1@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
LU-17351 ldiskfs: RHEL 9.3 ldiskfs server Updated patch series for el9.3 needs an updated ext4-data-in-dirent Test-Parameters: trivial env=SANITY_EXCEPT="906" \ mdtcount=4 mdscount=2 \ clientdistro=el9.3 serverdistro=el9.2 testlist=sanity Test-Parameters: trivial mdtcount=4 mdscount=2 \ clientdistro=el9.2 serverdistro=el9.3 testlist=sanity Test-Parameters: optional clientdistro=el9.3 serverdistro=el9.3 \ testgroup=full-part-1 Test-Parameters: optional clientdistro=el9.3 serverdistro=el9.3 \ testgroup=full-part-2 Test-Parameters: optional clientdistro=el9.3 serverdistro=el9.3 \ testgroup=full-part-3 HPE-bug-id: LUS-12050 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Signed-off-by: Jian Yu <yujian@whamcloud.com> Change-Id: Iac9731570422c57ef494602b1a40ac0b3d87d991 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53394 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Yang Sheng <ys@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: xinliang <xinliang.liu@linaro.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16502 lutf: add proper config option and fix bugs LUTF did not have a proper configuration option. Since no message was printed at configure time, this made it hard to debug why LUTF was not being built. Fix a few minor bugs in headers that prevented shared libraries from being `import`ed by python. Fix a small Clang error in liblutf_agent.c. Test-Parameters: @lnet Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I6680b203bef08b7afa326a1cbe30c96b5c29e95c Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53200 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Frank Sehr <fsehr@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16967 build: Separate lnet LND rpm packaging Enable separate packaging of lnet lnd kernel modules into separate packages. Use --with multiple_lnds to enable separate packages for lnet lnds: [always builds]: kmod-lustre-lnet-socklnd for socklnd.ko --with o2ib: for kmod-lustre-lnet-in-kernel-o2iblnd ko2iblnd.ko -> in-kernel-ko2iblnd.ko --with mofed: kmod-lustre-lnet-o2iblnd for ko2iblnd.ko --with kfi: kmod-lustre-lnet-gnilnd for kgnilnd.ko --with gni: kmod-lustre-lnet-kfilnd for kkfilnd.ko Test-Parameters: trivial HPE-bug-id: LUS-11711 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: If2dace7ced96be2a2194f66362e9419b017c625f Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51692 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Frank Sehr <fsehr@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16967 build: Add in-kernel-ko2iblnd driver Add in-kernel-ko2iblnd.ko for users of in-kernel OFED and only build ko2iblnd.ko if an external OFED is available This allows for building and packaging both an external (MOFED or OPA) o2ib driver and an in-kernel o2ib driver. Packaging rules will be written so that only enable one of the o2iblnd drivers can be installed. In the case of the in-kernel-ko2iblnd.ko driver a symlink named ko2iblnd.ko will be created to point to the in-kernel based o2ib driver which allows for a reasonable migration path for the majority of users. It is useful for dist build and test to be able to build both in-kernel IB and external OFED in the same build. This also means there would be some install/configure adjustments that ought to have some discussion. Test-Parameters: trivial HPE-bug-id: LUS-11711 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I8105fad0b20c36705d7e14e3ae976bf3d81e9f1b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51915 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Frank Sehr <fsehr@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17275 kernel: RHEL 8.9 client and server support This patch makes changes to support RHEL 8.9 release with kernel 4.18.0-513.5.1.el8_9 for Lustre client and server. Test-Parameters: trivial fstype=ldiskfs mdtcount=4 mdscount=2 \ clientdistro=el8.9 serverdistro=el8.8 testlist=sanity Test-Parameters: trivial fstype=zfs mdtcount=4 mdscount=2 \ clientdistro=el8.9 serverdistro=el8.8 testlist=sanity Test-Parameters: trivial fstype=ldiskfs mdtcount=4 mdscount=2 \ clientdistro=el8.8 serverdistro=el8.9 testlist=sanity Test-Parameters: trivial fstype=zfs mdtcount=4 mdscount=2 \ clientdistro=el8.8 serverdistro=el8.9 testlist=sanity Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \ testgroup=full-part-1 Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \ testgroup=full-part-2 Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \ testgroup=full-part-3 Change-Id: Ia3672d134534b877bb6aaffb4cea0339bc55974f Signed-off-by: Jian Yu <yujian@whamcloud.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53071 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Yang Sheng <ys@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17065 build: Remove snmp support Last patched in 2012 and not well tended to before that, snmp support can probably be removed. Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I1831aed42e560531e57a6ff8aa978f3e5286fd44 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52194 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16962 build: cleanup configure messages Take advantage of LB2_MSG_LINUX_TEST_RESULT to cleanup the remaining configure checks. Re-order checking of OpenSSL support so checking message and result are not split. Test-Parameters: trivial HPE-bug-id: LUS-11709 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I2880e2b50f4cc79106201c241fe7c078e5d8c37e Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51857 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Chris Horn <chris.horn@hpe.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: xinliang <xinliang.liu@linaro.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16976 ldiskfs: add support for openEuler 22.03 SP2 Add ldiskfs server support for oe2203sp2. Sync with ldiskfs-5.14-rhel9.2.series adding missing patches. Also refine openEuler lbuild scripts. Change-Id: I91841a7140a9f8f3182a4a329b9f04639a85e94d Test-Parameters: trivial Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51753 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 misc: update headers in config, debian, rpm Update the file header to have the SPDX license and use the standard format. Fix minor style issues with comments in a few files. Remove `dnl` from m4 files. Files that are uncertain are left as NOASSERTION for the license identifier. This makes no claim about the file. It is used to track files so they can be addressed later. https://spdx.github.io/spdx-spec/v2-draft/package-information/#75-package-supplier-field Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I212ce05a4292bbb0d71372d9d75880ce45a219f3 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52106 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16961 clang: plugins and build system integration Clang has a plugin system. Compiler extensions can be created by making a shared library and loading it via the "-fplugin" options. This makes it simple to implement custom warnings and static analyzers. This patch adds a plugin to detect functions that should have been made static. This plugin has been run over the majority of the Lustre tree and patches have been submitted for all warnings. The plugin did not return any false positives in my testing. It also add the "--enable-compiler-plugins" configure option, which automatically builds and sets up the in-tree C compiler plugins. The option force-enables the plugin regardless of which compiler is in use. This behavior could be changed if there is ever a need to support GCC specific plugins. Also, add the configure checks needed to support building C++ in the Lustre tree. Clang and GCC plugins (and the compilers themselves) are written in C++. The license for the plugin mirrors that of the LLVM project itself. This leaves the door open for contributing this plugin upstream in the future. This isn't being upstreamed now because it lacks any significant user community. Hence, the plugin does not appear to meet the requirements for upstreaming based on https://clang.llvm.org/get_involved.html. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I747ed91b53e765cc58e91a3eb9ec6c12b9908a96 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51659 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16906 build: Server for newer SUSE 15 SP3 kernels Update the SUSE 15 SP3 server support for newer kernels including LTSS series kernels. Add a new ldiskfs patch series for updated SUSE 15 SP3 kernels with a updated ext4-pdirop.patch Test-Parameters: trivial HPE-bug-id: LUS-11676 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I0acf81abfcc71a64dc09a344a9231d86a44f193e Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51338 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16962 build: parallel header checks Add LB2_CHECK_LINUX_HEADER_SRC and LB2_CHECK_LINUX_HEADER_RESULT macros to use for running header checks in parallel. Migrate (most) header checks to parallel and run a subset early as the results of those tests are required by other configure tests. Test-Parameters: trivial HPE-bug-id: LUS-11710 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: Ia765261179d25e96912e65e31c81824b4507e604 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51673 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
LU-16957 build: Improve parallel --config-cache The parallel build should consider the configure cache before adding tests to the parallel build pass. Track the number of compile tests needed, skip the make when no build tests are needed. Also unify libcfs, core, and ldiskfs build passes to a single step. Configure timings vs master master master w/cache | patch patch w/cache -------------- --------------- | --------------- ---------------- real 1m3.493s real 0m34.024s | real 1m3.903s real 0m8.404s user 1m34.587s user 1m16.547s | user 1m37.191s user 0m4.292s sys 0m35.119s sys 0m22.687s | sys 0m35.297s sys 0m5.514s Test-Parameters: trivial HPE-bug-id: LUS-11706 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I6696b350e8315190a67c1463435b18a87d45813e Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51637 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Chris Horn <chris.horn@hpe.com>
LU-16053 build: Update zfs configure checks From Brian Behlendorf <behlendorf1@llnl.gov>: update dmu_*_by_dnode checks Provided as a feature since ZFS 0.7.0, convert to a fatal configure error when unavailable. update zap_*_by_dnode checks Provided as a feature since ZFS 0.7.0, convert to a fatal configure error when unavailable. update multihost protection check Provided as a feature since ZFS 0.7.0, convert to a fatal configure error when unavailable. Drop the compatibility code required to support OpenZFS releases older than 0.7.0. update userobj accounting check Provided as a feature since ZFS 0.7.0, convert to a fatal configure error when unavailable. update dmu_prefetch() check Provided since at least ZFS 0.7.0, convert to a fatal configure error when unavailable. update dmu_object_alloc_dnsize() check Provided since at least ZFS 0.7.0, convert to a fatal configure error when unavailable. update spa_maxblocksize() check Provided since at least ZFS 0.7.0, convert to a fatal configure error when unavailable. update dsl_pool_config_enter/exit check Convert to a fatal configure error, these functions have been provided since at least ZFS 0.7.x. replace sa_spill_block() check The sa_spill_block() function was removed after the ZFS 0.6.x release. Replace the check with one for use zio_buf_alloc/free which have been available since 0.7.x. The dsl_sync_task_do_nowait() function has not been provided by since the 0.6.x releases. Furthermore, the results of this check are unused by Lustre so let's just remove it. Test-Parameters: trivial Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I3c1597e56100961178f9001e918ffb9aa3558706 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48089 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16405 osd: lookup cache MDT may need to re-lookup just checked names (after locking). introduce a trivial tiny per-thread cache in OSD in order to make such a repeating lookup cheap. the original issue is that ext4_add_entry() doesn't really check for possible duplicate (that would be expensive as a whole 4K block must be scanned). important: the cache is reset upon request processing completion as we don't update iversion on a disk (due to conflict with VBR). Fixes: 79acb9a9e7 ("LU-10235 mdt: mdt_create: check EEXIST without lock") Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: I40c3ee702f7895c3bda00b380f904cd587e0a1c4 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50521 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>