LU-16969 build: check that pkg-config is installed PKG_CHECK_MODULES macro fails in very annoying to debug ways. Often, this will fail with: syntax error near unexpected token `LIBNL3,' ` PKG_CHECK_MODULES(LIBNL3, libnl-genl-3.0 >= 3.1)' and provide no indication that the real error is that pkg-config is not installed. Adding an explicit check for pkg-config will make the error more self-evident. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Signed-off-by: James Simmons <jsimmons@infradead.org> Change-Id: Ic2ee8e4c3ec3fa2e03c5ece03e6a9ce335133578 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51716 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16518 build: llvm/clang support Other projects, notably Linux, have build support for LLVM and Clang via special environment variables. This is implemented for Lustre, in the style of: https://www.kernel.org/doc/html/latest/kbuild/llvm.html Instances in which GCC is explicitly called are replaced by the use of $CC. The proper environment variables as passed to make invocations as needed. All checks which influence global compiler and toolchain settings are collected in 'config/lustre-toolchain.m4'. A configure option is added to disable the strict error flags that are passes to the C compiler by default. CFLAGS and EXTRA_CFLAGS are made to work in the typical way. Having fine grained control over compiler options makes experimenting with Clang smoother. Some compile checks in 'lustre-core.m4' have been improved by using unused variables and explicitly setting the compile flag to be used during the test. This also sets the execute bit on autogen.sh. Tested with: Linux (mainline) - 5.15.94 openZFS - 2.1.99 Lustre (latest master) - 2.15.55 CentOS - 8.5 Clang (default on CentOS) - 12.0.1 Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Ia8654c22fa8fca7bfb96c545ac144a1d3737fa00 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50063 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 build: add SPD/GPL license to build files Update the file header to have the SPDX license and use the standard format. Convert spaces to tabs. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Id02218aa5b435bc0de96a39d3daa53a83a51c857 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50347 Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-15838 autoconf: fix use of obsolete macros This patch fixes the following warnings when using autoconf 2.71: configure.ac:2: warning: AC_INIT: not a literal: "m4_esyscmd(sh -c "./LUSTRE-VERSION-GEN | tr -d '\n'")" configure.ac:10: warning: The macro `AC_CANONICAL_SYSTEM' is obsolete. configure.ac:16: warning: The macro `AC_PROG_LIBTOOL' is obsolete. configure.ac:24: warning: The macro `AC_HELP_STRING' is obsolete. Like m4_esyscmd, macro m4_esyscmd_s (introduced in autoconf 2.64) expands to the result of running command in a shell. The difference is that any trailing newlines are removed. Since autoconf 2.50, macro 'AC_CANONICAL_TARGET' has been the new name of 'AC_CANONICAL_SYSTEM': AU_ALIAS([AC_CANONICAL_SYSTEM], [AC_CANONICAL_TARGET]) Since autoconf 2.58, macro 'AS_HELP_STRING' has been added to replace 'AC_HELP_STRING'. Since libtool 2.0, new 'LT_INIT' interface has been added to replace 'AC_PROG_LIBTOOL'. Change-Id: I3c06c21460d7a2cf643fe825e72a26a5416609cf Signed-off-by: Jian Yu <yujian@whamcloud.com> Reviewed-on: https://review.whamcloud.com/47288 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15967 build: configure script does not check for required build tools - added check for flex and bison - added requirement for build kernel modules Change-Id: I4f4f19ea44f3cd8f69482d950970bf701e81f7ec Signed-off-by: Alex Deiter <alex.deiter@gmail.com> Reviewed-on: https://review.whamcloud.com/47744 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14198 build: Use pax tar format for build of dist-targets The tar ustar format used during build of dist-* targets (which is used when building e.g. rpms and debs) has several limitations and will prevent e.g. users with uid > 2097151 (2^21) from building these targets (or from passing the autoconf test for tar). This commit changes format from ustar to POSIX.1-2001/pax which will remove this and several other limits (with path-length being one relevant example, see LU-12078). Signed-off-by: Daniel Ahlin <ahlin@google.com> Change-Id: Ic66ca696ede2e359a04c179c6d630baacaa9bcb1 Reviewed-on: https://review.whamcloud.com/40915 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Minh Diep <mdiep@whamcloud.com>
LU-11304 misc: update all url links to whamcloud Even old links could redirect to whamcloud automatically, We'd better update to use new whamcloud links to avoid any further confusions. Test-parameters: trivial Change-Id: Ida7161a062d822141bf0c1fdf20b2098a21ea9e7 Signed-off-by: Wang Shilong <wshilong@ddn.com> Reviewed-on: https://review.whamcloud.com/33094 Reviewed-by: Peter Jones <pjones@whamcloud.com> Tested-by: Jenkins Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <hpdd-maloo@intel.com>
LU-5541 build: move libcfs and liblustreapi over to libtool Change libcfs into a convenience library using libtool. This allows use to embbed libcfs library into both liblnetconfig and liblustreapi so their is no longer a need to link applications to libcfs.a anymore. With this change we need migrate liblustreapi to libtool. libtool knows how to build both static and dymanic libraries for liblusteapi, so no need to hack the Makefile. As two added benefits, the utilities will now use the dynamic version, thus reducing their footprint, and calling make twice in a row won't rebuild objects already built. Test-Parameters: trivial Change-Id: Icc1e5d42df503b9bf393396fe09f4e4f1f242486 Signed-off-by: frank zago <fzago@cray.com> Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/30562 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-7699 build: Convert lustre_ver.h.in into a static .h file The AC_LUSTRE_{MAJOR,MINOR,PATCH,FIX} autoconf output values are no longer necessary. We can instead use AC_DEFINE to set the LUSTRE_{MAJOR,MINOR,PATCH,FIX} CPP macros directly. As a result, the lustre_ver.h file no longer needs to be generated by autoconf and it can be made into a static file. At the same time we set LUSTRE_VERSION_STRING, even though it is not strictly needed, to avoid additional divergence from the client code in the upstream Linux kernel staging area. Change-Id: I6b0364a7af1869beb6acedc63c034565af00ef06 Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/18111 Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: James Simmons <uja.ornl@yahoo.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-7699 build: Replace version_tag.pl with LUSTRE-VERSION-GEN This patch overhauls how version numbers are handled. The biggest change is removing the two independent places where versions were being generated (LB_BUILDID and lustre_tag*.pl) with a single script named LUSTRE-VERSION-GEN at the top level of the tree. LUSTRE-VERSION-GEN is based on the GIT-VERSION-GEN script that git's own source code build system uses. LUSTRE-VERSION-GEN generates the lustre version string from either git, or if git is not available from the LUSTRE-VERSION-FILE file, or if _that_ is not available, its own internal DEFAULT_VERSION default version number. This allows building the lustre code under all reasonable conditions. The new method uses LUSTRE-VERSION-GEN to supply the correct full lustre version (which may contain git --describe information) to AC_INIT. This means that the autotools actually know the correct version number. Some benefits of this are: * We can use standard {AC_}PACKAGE_VERSION macros and variables instead of inventing our own * ./configure --version and other commands will list the correct version of Lustre * The lustre tarball created by "make dist" will contain the full version number The lustre/autoconf/lustre-version.ac file was replaced with a new config/lustre-version.m4 file. We no longer need to work with m4-only macros prefore AC_INIT is called, so the new file defines a LUSTRE_VERSION_VARIABLES function for us to call from configure.ac. The new lustre-version.m4 derives the derive LUSTRE_{MAJOR,MINOR,PATCH,FIX} from AC_PACKAGE_VERSION rather than being yet another source of version numbers. The LUSTRE_VER_ALLOWED_OFFSET and LUSTRE_CLI_VER_OFFSET_WARN m4 macros were dropped, and the OBD_OCD_VERSION strings stored directly in lustre_ver.h.in instead. LB_CHECK_VERSION was removed (no longer necessary). AC_LUSTRE_VERSION_STRING was drop. It is no longer necessary now that we can set LUSTRE_VERSION_STRING directly to @PACKAGE_VERSION@ in lustre_ver.h.in. The DOWNSTREAM_RELEASE configure argument is deprecated and all related code removed. The lustre/doc/VERSIONING documenation is relocated to Documentation/versioning.txt and updated to reflect the new versioning method and practices. The META file and BUILDID variables and related code are removed since they are no longer used under the new versioning method. Note that this patch makes lustre only look at the dotted version tags in git, so we can finally stop using three to five tags for every release. Now we can just use one clear tag. The tags with underscores are obsolete. Change-Id: I7196f92ed4975776cc8e73fd3ebf436b5921f83e Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/18107 Tested-by: Jenkins Reviewed-by: James Simmons <uja.ornl@yahoo.com> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5919 build: Make "make rpms" use "make srpm" output We rework the rpms and srpm Make targets. The srpm target generates the source rpm package much as it did before, but it puts the results in a known temporary rpm build tree. This allows the "make rpms" target to simply depend on the srpm target, and then build the binary rpms from the source rpm. Previously the rpm target _also_ generated a source rpm package, which could be redudant, and failed to verify that the source rpm package was itself buildable. Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com> Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Change-Id: I9b2e83e000aca62b5d4841e2ebe9e91fa1db300f Reviewed-on: http://review.whamcloud.com/12961 Tested-by: Jenkins Reviewed-by: James Simmons <uja.ornl@yahoo.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Bob Glossman <bob.glossman@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5306: Update download and other whamcloud URLs We are updating *.whamcloud.com URLs to *.hpdd.intel.com URLs, as well as downloads.lustre.org to downloads.hpdd.intel.com. Signed-off-by: Joshua Kugler <joshua.kugler@intel.com> Change-Id: I6ebd45f8bb1f905692ca992617805ef225e09849 Reviewed-on: http://review.whamcloud.com/11169 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Reviewed-by: Peter Jones <peter.a.jones@intel.com>
LU-4606 utils: Mount uses so for backing fstype Make mount/mkfs/tunefs functionality a per backing FSTYPE shared object that is loadable, and seperately buildable and installable in the appropriate lustre-osd-FSTYPE rpm. For ZFS this allows the shared object to be directly linked to libzfs and thus remove the hardcoding of a libzfs.so version in the lustre source. This also has the benefit of making lustre-osd-zfs explicitly requires libzfs (which it needs to do mount/mkfs operations). Adjust build system to allow building of loadable module. Including requiering libtool as part of the build. Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com> Change-Id: I8fc0f41161dc08405c1f951f12e675f364ff62c3 Reviewed-on: http://review.whamcloud.com/10193 Tested-by: Jenkins Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3462 ldiskfs: Subsume ldiskfs's build system into lustre Remove ldiskfs's independant build system (configure, spec file, etc.) and merge its build into lustre's build system. Move the ldiskfs/ldiskfs directory up one level. The extra directory level is no longer needed without the independant ldiskfs build system. Add a make dependency on ldiskfs's "sources" target in ldiskfs/autoMakefile.am rather than explicitly launching a separate make instance from the top level autoMakefile.am. Remove unused "sources" targets in libcfs, lnet, and lustre/obdclass. Improve the handling of the --disable-ldiskfs option, and make it interact with other configure options more cleanly. Change-Id: Id0787f27269e938032e480c7c627327c0adf2672 Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/6850 Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Minh Diep <minh.diep@intel.com> Reviewed-by: Brian J. Murrell <brian.murrell@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1199 build: Clean out the build directory Clean up the build directory. Move in the direction of reserving "build/" for the special-purpose Makefile that lives there. Eventually we could rewrite the autoconf tests to eliminate that Makfile, and the build directory could disappear altogether (after finding homes for anything else that is left). The autoconf m4 file move into a top level "config" directory. Most other things that have moved are put in the new "contrib" top-level directory. For instance, "contrib/lbuild" contains all of the lbuild-related files, and "contrib/git-hooks" contains the git hooks for lustre developers. Most of the moved files were unchanged, however the lbuild scripts needed some tweaking to deal with the new location. Because of the way that Intel's build farm and git hooks expect to find certain files in fixed locations I have had to leave a few symlinks in place that point to the new locations. Change-Id: I04dc529d4f4060b892e1e4eaa8613bbc3337c414 Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/5035 Reviewed-by: Brian J. Murrell <brian.murrell@intel.com> Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1032 build: Support --disable-maintainer mode We add "AM_MAINTAINER_MODE([enable])" to all configure scripts to allow us to use --disable-maintainer-mode. By default, without the AM_MAINTAINER_MODE macro, autotools "maintainer mode" is enabled. By specifying "enable" we maintain our previous default behaviour. Change-Id: I88366ad658795145af80ed96c6e708c385799ffa Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/5423 Tested-by: Hudson Reviewed-by: Faccini Bruno <bruno.faccini@intel.com> Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Prakash Surya <surya1@llnl.gov> Reviewed-by: Brian J. Murrell <brian.murrell@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1199 build: Simplify autotools version checks Simplify the check for minumum autoconf and automake versions by using standard macros in configure.ac rather than using shell functions in autogen.sh. Note that I am bumping the minimum automake version from 1.9 to 1.10. Change-Id: I009c316a54d8d517abea0d1808dc244e2353b2aa Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov> Reviewed-on: http://review.whamcloud.com/4693 Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Brian J. Murrell <brian.murrell@intel.com> Reviewed-by: Minh Diep <minh.diep@intel.com>
LU-2068 build: Disable obdfilter with osd-zfs This change introduces a few different cleanup and build system changes. Most notably, the obdfilter modules is now disabled and the mkfs utilities does not call into ldiskfs helpers when ldiskfs is disabled. Apart from that, the other changes are various minor cleanup and dead code removal. Signed-off-by: James Simmons <uja.ornl@gmail.com> Signed-off-by: Prakash Surya <surya1@llnl.gov> Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com> Change-Id: Idcfda525e4b76269c2cf17c1b3232033b92fc8a8 Reviewed-on: http://review.whamcloud.com/4159 Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
LU-2068 build: Respect "--without-ldiskfs" option The "--without-ldiskfs" configure option was not properly being respected by a few of the Makefiles. This change is an attempt to remedy this issue. Now, one should be able to disable building ldiskfs and the related lvfs and osd code with "--without-ldiskfs". Signed-off-by: Prakash Surya <surya1@llnl.gov> Change-Id: I502e125639a60f62a37e3f95ecc943dedc37f5ea Reviewed-on: http://review.whamcloud.com/4148 Tested-by: Hudson Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1869 build: gcc/ar utility can vary - ar utility can vary depeding on build platform. - do not call gcc directly, use $(CC) Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: Ia60752b40531886d805a095c9f571e565bcd0e88 Reviewed-on: http://review.whamcloud.com/3917 Tested-by: Hudson Tested-by: Maloo <whamcloud.maloo@gmail.com> Reviewed-by: Liu Xuezhao <xuezhao.liu@emc.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>