LU-16967 build: Separate lnet LND deb packaging Enable separate packaging of lnet lnd kernel modules into separate packages with build profile multiple-lnds: lustre-lnet-module-socklnd for socklnd.ko lustre-lnet-module-gnilnd for kgnilnd.ko, profile gnilnd lustre-lnet-module-kfilnd for kkfilnd.ko, profile kfilnd lustre-lnet-module-o2iblnd for o2iblnd.ko, profile ext_o2ib lustre-lnet-module-in-kernel-o2iblnd for ko2iblnd.ko, profile int_o2ib Test-Parameters: trivial HPE-bug-id: LUS-11711 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I3a5ca03fa410238f66083289db0899c8b4bfab5c Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52397 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: Frank Sehr <fsehr@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17000 contrib: script to prepare coverity builds Add script 'coverity-run' to semi-automate running and submitting Coverity builds for Lustre. This should make it much easier to reproducibly submit builds to Coverity - and serve as an example of how the Coverity build process works. It should also provide more transparency in how builds are being prepared for Coverity. Add a Vagrantfile for the Vagrant VM used during the build process. Update in-tree Documentation. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I050b10d9df0e4e4c1b8bcc91a3c296c11f27ffef Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53400 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16502 python: improve support for virtual environments Python virutal environments make it easy to install the correct Python packages isolated from the rest of the system. https://docs.python.org/3/library/venv.html .venv is added to .gitignore and a simple virtual environment example has been added to the README. This patch collects all of the requirements for various scripts in the Lustre tree and consolidates them in a top level requirements.txt. lu_object.py spacing was fixed due to parsing errors. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I69d074e9ba50022817bd243fb82d004366ab6adf Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53209 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16967 build: Add in-kernel-ko2iblnd driver Add in-kernel-ko2iblnd.ko for users of in-kernel OFED and only build ko2iblnd.ko if an external OFED is available This allows for building and packaging both an external (MOFED or OPA) o2ib driver and an in-kernel o2ib driver. Packaging rules will be written so that only enable one of the o2iblnd drivers can be installed. In the case of the in-kernel-ko2iblnd.ko driver a symlink named ko2iblnd.ko will be created to point to the in-kernel based o2ib driver which allows for a reasonable migration path for the majority of users. It is useful for dist build and test to be able to build both in-kernel IB and external OFED in the same build. This also means there would be some install/configure adjustments that ought to have some discussion. Test-Parameters: trivial HPE-bug-id: LUS-11711 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I8105fad0b20c36705d7e14e3ae976bf3d81e9f1b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51915 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Frank Sehr <fsehr@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15581 misc: update .gitignore files Ignore the binary for check_iam utility in lustre/utils. Also, ignore more files for commit messages. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I5b11dc2d2f3761f778549a121ac940edeeb70980 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49632 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: jsimmons <jsimmons@infradead.org>
LU-16359 build: RHEL use Module.symvers during find-provides find-provides fails to find module versions on newer kernels. The generated Module.symvers is always generated and correct. Install it to the well known location BUILDROOT use it to generate provides and ignore it for installation. Create a new find-provides and find-provides.ksyms for lustre based on the one provided by the redhat-rpm-config package using Module.symvers to supply the symbol versions instead of extracting symbol versions from the .ko files. Test-Parameters: trivial HPE-bug-id: LUS-11383 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I01c3b3692e6a2a6be86a6930eaead9df75147f90 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49302 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14651 ldiskfs: add 5.11 kernel support The Ubuntu 20.04.3 LTS moved to the 5.11 kernel. Support for this kernel is a small step from the 5.10 kernel support. This patch adds these small changes to support ldiskfs. Test-Parameters: trivial Change-Id: I3055736658b628fe79a6a9fc20ac01e7e1597630 Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/47900 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13344 libcfs: Abstract proc_fs with proc_ops Linux 5.6 introduces proc_ops with v5.5-8862-gd56c0d45f0e2 proc: decouple proc from VFS with "struct proc_ops" Map proc_ops and it's members to file_operations and the appropriate members for older kernels. One remaining 'PROC_OWNER()' macro is left to deal with proc_ops being unable to sensibly map the owner member. Test-Parameters: trivial HPE-bug-id: LUS-8589 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I3d8940a91b331c4f6bb31a9432194cc082c9cecd Reviewed-on: https://review.whamcloud.com/37873 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10973 lnet: infrastructure to build the LUTF Add flags to turn on/off LUTF building. Modify the gitignore to ignore .i files which are the swig interface files used to create python callable APIs from C APIs m4 files to search and find python and swig installations needed for building the LUTF. Test-Parameters: trivial Signed-off-by: Amir Shehata <ashehata@whamcloud.com> Change-Id: Idbd23dc457c95425edbf88755ae261ff4de6b0c9 Reviewed-on: https://review.whamcloud.com/38084 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12634 build: kbuild changes in 5.3 drop subdir-m Several changes in kbuild affect the way external modules can be built. In the Linux 5.3-rc4 series subdir-m has been removed. Linux commit: c07d8d47bca1b325102fa2be3a463075f7b051d9 Test-Parameters: trivial Cray-bug-id: LUS-7689 Signed-off-by: Shaun Tancheff <stancheff@cray.com> Change-Id: Id1f248ac4ccdee8d2a2d177b4fdff4444d2084d1 Reviewed-on: https://review.whamcloud.com/35786 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10997 build: add files to .gitignore To avoid by accident adding files created during the build process add them to the .gitignore. For Ubuntu 18 we add in .cache.mk and *.o.ur-safe which only show up on that platform Test-Parameters: trivial Change-Id: Ie1329e765f080cbdac1bd3efdd63f83a65d45989 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/32304 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Bob Glossman <bob.glossman@intel.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5541 build: move libcfs and liblustreapi over to libtool Change libcfs into a convenience library using libtool. This allows use to embbed libcfs library into both liblnetconfig and liblustreapi so their is no longer a need to link applications to libcfs.a anymore. With this change we need migrate liblustreapi to libtool. libtool knows how to build both static and dymanic libraries for liblusteapi, so no need to hack the Makefile. As two added benefits, the utilities will now use the dynamic version, thus reducing their footprint, and calling make twice in a row won't rebuild objects already built. Test-Parameters: trivial Change-Id: Icc1e5d42df503b9bf393396fe09f4e4f1f242486 Signed-off-by: frank zago <fzago@cray.com> Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/30562 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-8789 build: add files to .gitignore Exclude files that are generated during the "make" build process. Test-Parameters: trivial Change-Id: I5a573d7842fec94616de5d016cc5aab99e813193 Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com> Reviewed-on: https://review.whamcloud.com/23540 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: James Simmons <uja.ornl@yahoo.com> Reviewed-by: John L. Hammond <john.hammond@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-7699 build: Replace version_tag.pl with LUSTRE-VERSION-GEN This patch overhauls how version numbers are handled. The biggest change is removing the two independent places where versions were being generated (LB_BUILDID and lustre_tag*.pl) with a single script named LUSTRE-VERSION-GEN at the top level of the tree. LUSTRE-VERSION-GEN is based on the GIT-VERSION-GEN script that git's own source code build system uses. LUSTRE-VERSION-GEN generates the lustre version string from either git, or if git is not available from the LUSTRE-VERSION-FILE file, or if _that_ is not available, its own internal DEFAULT_VERSION default version number. This allows building the lustre code under all reasonable conditions. The new method uses LUSTRE-VERSION-GEN to supply the correct full lustre version (which may contain git --describe information) to AC_INIT. This means that the autotools actually know the correct version number. Some benefits of this are: * We can use standard {AC_}PACKAGE_VERSION macros and variables instead of inventing our own * ./configure --version and other commands will list the correct version of Lustre * The lustre tarball created by "make dist" will contain the full version number The lustre/autoconf/lustre-version.ac file was replaced with a new config/lustre-version.m4 file. We no longer need to work with m4-only macros prefore AC_INIT is called, so the new file defines a LUSTRE_VERSION_VARIABLES function for us to call from configure.ac. The new lustre-version.m4 derives the derive LUSTRE_{MAJOR,MINOR,PATCH,FIX} from AC_PACKAGE_VERSION rather than being yet another source of version numbers. The LUSTRE_VER_ALLOWED_OFFSET and LUSTRE_CLI_VER_OFFSET_WARN m4 macros were dropped, and the OBD_OCD_VERSION strings stored directly in lustre_ver.h.in instead. LB_CHECK_VERSION was removed (no longer necessary). AC_LUSTRE_VERSION_STRING was drop. It is no longer necessary now that we can set LUSTRE_VERSION_STRING directly to @PACKAGE_VERSION@ in lustre_ver.h.in. The DOWNSTREAM_RELEASE configure argument is deprecated and all related code removed. The lustre/doc/VERSIONING documenation is relocated to Documentation/versioning.txt and updated to reflect the new versioning method and practices. The META file and BUILDID variables and related code are removed since they are no longer used under the new versioning method. Note that this patch makes lustre only look at the dotted version tags in git, so we can finally stop using three to five tags for every release. Now we can just use one clear tag. The tags with underscores are obsolete. Change-Id: I7196f92ed4975776cc8e73fd3ebf436b5921f83e Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/18107 Tested-by: Jenkins Reviewed-by: James Simmons <uja.ornl@yahoo.com> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-7174 build: add build by products to .gitignore While testing patches other non-patch related build by products show up with git status. To avoid adding these by accident place thes by product files in the proper .gitignore files. Change-Id: Ie2df9c2c7fd19c95e2b990d93db623826ee82c24 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: http://review.whamcloud.com/16458 Reviewed-by: Bob Glossman <bob.glossman@intel.com> Tested-by: Jenkins Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-7042 lnet: Handle OFED 3.18 packaging definitions Starting with OFED 3.18 the OFED package started to use config.h provided by autoconf tool. This lead to clash between PACKAGE_* macros which are defined in OFED and Lustre headers. Also dealing with kernels that have lustre enabled already required to undefine the common macros that used by in-kernel Lustre client and this sources. This fix undefine all symbols that are generated by Lustre autoconf to avoid conflicts with kernel defines or OFEDs. They are undefined right before new definition in config.h. The list of symbols to undef is automatically generated by autoconf and should not be extended in the future. Also undefine clashed macros in autoconf checks. Change-Id: I0d93adac19573328e905ba536db0dbd842ed2aab Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com> Reviewed-on: http://review.whamcloud.com/16418 Tested-by: Jenkins Reviewed-by: James Simmons <uja.ornl@yahoo.com> Reviewed-by: Bob Glossman <bob.glossman@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4606 utils: Mount uses so for backing fstype Make mount/mkfs/tunefs functionality a per backing FSTYPE shared object that is loadable, and seperately buildable and installable in the appropriate lustre-osd-FSTYPE rpm. For ZFS this allows the shared object to be directly linked to libzfs and thus remove the hardcoding of a libzfs.so version in the lustre source. This also has the benefit of making lustre-osd-zfs explicitly requires libzfs (which it needs to do mount/mkfs operations). Adjust build system to allow building of loadable module. Including requiering libtool as part of the build. Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com> Change-Id: I8fc0f41161dc08405c1f951f12e675f364ff62c3 Reviewed-on: http://review.whamcloud.com/10193 Tested-by: Jenkins Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1032 build: Add Lustre DKMS spec file Add a lustre-dkms.spec file which can be used to distribute dkms style Lustre modules. The spec file is originally based on the generic dkms template and the default behavior is as follows: * Disable ldiskfs osd support. The ldiskfs packages currently cannot be built reliably against arbitrary kernels and are therefore disabled by default. * Enable zfs osd support. ZFS dkms packages are hosted at http://archive.zfsonlinux.org/{epel,fedora}/{release}/ and are compatible once LU-3117 is merged in to the Lustre source. * Some of the default Lustre build options can be changed by setting parameted in the /etc/sysconfig/lustre config file. Going forward the options can be extended as needed. The currently supported options are: * LUSTRE_DKMS_DISABLE_CDEBUG=y|N * LUSTRE_DKMS_DISABLE_TRACE=y|N * LUSTRE_DKMS_DISABLE_ASSERT=y|N * LUSTRE_DKMS_DISABLE_STRIP=y|N * A build target was not added for the lustre-dkms.spec file. To create lustre dkms packages you must manually invoke rpmbuild. ./configure --enable-dist make dist rpmbuild -bs lustre-dkms.spec lustre-x.y.z.tar.gz rpmbuild --rebuild lustre-dkms-x.y.z-r.dist.src.rpm Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Bruno Faccini <bruno.faccini@intel.com> Change-Id: I870f362b8948d5cd28a8dccd98b565e38ad2da7c Reviewed-on: http://review.whamcloud.com/6019 Tested-by: Jenkins Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
LU-2771 dlmlock: compress out unused space * lustre/include/lustre_dlm.h: Remove all bit fields and the unused weighing callback procedure. respell LDLM_AST_DISCARD_DATA as LDLM_FL_AST_DISCARD_DATA to match other flags. * .gitignore: ignore emacs temporary files * autogen.sh: rebuild the lock bits, if autogen is available. * contrib/bit-masks/lustre_dlm_flags.def: define the ldlm_lock flags * contrib/bit-masks/lustre_dlm_flags.tpl: template for emitting text * contrib/bit-masks/Makefile: construct the .c and .h files The .c file is for constructing a crash extension and is not preserved. * contrib/bit-masks/.gitignore: ignore built products * lustre/contrib/wireshark/packet-lustre.c: use built files instead of local versions of the defines. In the rest of the modified sources, replace flag field references with bit mask references. * lustre/osc/osc_lock.c: removed osc_lock_weigh, too Signed-off-by: Bruce Korb <bruce_korb@xyratex.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com> Reviewed-by: Keith Mannthey <Keith.Mannthey@intel.com> Change-Id: I104522e4012deb0dd8b9d12c2d1f3fd3851708cb Reviewed-on: http://review.whamcloud.com/5312 Reviewed-by: Keith Mannthey <keith.mannthey@intel.com> Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: <bruce.korb@gmail.com>
LU-1617 build: ignore automatically generated files Do not list automatically generated files in "git status" output, so that it does not hide checkpatch.pl output in the commit message. Signed-off-by: Andreas Dilger <andreas.dilger@intel.com> Change-Id: Iab5c5be25de7dc54225fb2aa6540a4d75e500c1e Reviewed-on: http://review.whamcloud.com/6229 Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com> Reviewed-by: Keith Mannthey <keith.mannthey@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>