Whamcloud - gitweb
EX-6275 lustre: add lz4 and lz4hc kernel modules
authorSebastien Buisson <sbuisson@ddn.com>
Thu, 17 Nov 2022 07:31:45 +0000 (08:31 +0100)
committerAndreas Dilger <adilger@whamcloud.com>
Fri, 26 May 2023 10:24:52 +0000 (10:24 +0000)
commit8238c4e7fe0b9b2135f0932f93bbcd6a9be0d897
tree9ab3f0011190848696b62570562963595a6c4bd2
parentd07948213803f68adee3e1ce5a51dd7206529261
EX-6275 lustre: add lz4 and lz4hc kernel modules

lz4 and lz4hc kernel modules implement compression according to the
lz4 and lz4hc algorithms respectively, through the kernel Crypto API.
lz4 module provides 2 cipher drivers under the generic name 'lz4':
* lz4-lustre-generic of type compression
* lz4-lustre-scomp of type scomp
lz4hc module provides 2 cipher drivers under the generic name 'lz4hc':
* lz4hc-lustre-generic of type compression
* lz4hc-lustre-scomp of type scomp

lz4 and lz4hc kernel module sources are copied from linux v6.1-rc5,
and renamed to llz4.c and llz4hc.c respectively to avoid name
collisions. Use of vmalloc has been changed to kvmalloc since it is
faster in most cases. They implement the Crypto API interface, and
rely on the lz4/lz4hc kernel library for compression implementation.
They have been modified to grok a compression acceleration/level, as
read from the top 4 bits of the crypto_tfm flags, and pass it to the
underlying library.
The lz4/lz4hc library sources are also copied from linux v6.1-rc5 and
built statically, so lz4_compress, lz4_decompress and lz4hc_compress
sources have been "de-modulified", and EXPORT_SYMBOLs removed. Headers
have also been copied from linux v6.1-rc5 for consistency, and source
files modified to include the copied headers instead of the system
headers.
All aforementioned sources are located in the lustre/lz4 directory.

The lz4/lz4hc modules are built only if the kernel does not provide
them. This is verified by checking if the CONFIG_CRYPTO_LZ4 and
CONFIG_CRYPTO_LZ4HC kernel config options are defined.

This patch provides unit testing of several compression modules, such
as lz4, lz4hc, lzo and gzip. This is done via a new test kernel module
kcompr.ko, and new sanity test_84. File lustre/tests/kernel/kcompr.c
contains examples of how to call the compression/decompression
routines.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I74ce95ff18194c6052d291588d7e8c79596a7f23
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49208
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
26 files changed:
config/lustre-build.m4
debian/dkms.conf.in
debian/rules
lustre.spec.in
lustre/Makefile.in
lustre/autoMakefile.am
lustre/autoconf/lustre-core.m4
lustre/include/lustre_crypto.h
lustre/lz4/Makefile.in [new file with mode: 0644]
lustre/lz4/autoMakefile.am [new file with mode: 0644]
lustre/lz4/llz4.c [new file with mode: 0644]
lustre/lz4/llz4hc.c [new file with mode: 0644]
lustre/lz4/lz4.h [new file with mode: 0644]
lustre/lz4/lz4_compress.c [new file with mode: 0644]
lustre/lz4/lz4_decompress.c [new file with mode: 0644]
lustre/lz4/lz4defs.h [new file with mode: 0644]
lustre/lz4/lz4hc_compress.c [new file with mode: 0644]
lustre/lz4/scompress.h [new file with mode: 0644]
lustre/scripts/dkms.mkconf
lustre/scripts/lustre_rmmod
lustre/tests/AMSR_E_L3_DailyOcean_V05_20111003.hdf.bz2 [new file with mode: 0644]
lustre/tests/Makefile.am
lustre/tests/kernel/Makefile.in
lustre/tests/kernel/autoMakefile.am
lustre/tests/kernel/kcompr.c [new file with mode: 0644]
lustre/tests/sanity.sh