New Features 0.8.0
* Native encryption #5769 - The encryption property enables the
creation of encrypted filesystems and volumes. The aes-256-ccm
algorithm is used by default. Per-dataset keys are managed with zfs
load-key and associated subcommands.
* Raw encrypted 'zfs send/receive' #5769 - The zfs send -w option
allows an encrypted dataset to be sent and received to another pool
without decryption. The received dataset is protected by the original
user key from the sending side. This allows datasets to be efficiently
backed up to an untrusted system without fear of the data being
compromised.
* Device removal #6900 - This feature allows single and mirrored
top-level devices to be removed from the storage pool with zpool
remove. All data is copied in the background to the remaining
top-level devices and the pool capacity is reduced accordingly.
* Pool checkpoints #7570 - The zpool checkpoint subcommand allows
you to preserve the entire state of a pool and optionally revert back
to that exact state. It can be thought of as a pool wide snapshot.
This is useful when performing complex administrative actions which
are otherwise irreversible (e.g. enabling a new feature flag,
destroying a dataset, etc).
* Pool TRIM #8419 - The zpool trim subcommand provides a way to
notify the underlying devices which sectors are no longer allocated.
This allows an SSD to more efficiently manage itself and helps prevent
performance from degrading. Continuous background trimming can be
enabled via the new autotrim pool property.
* Pool initialization #8230 - The zpool initialize subcommand writes
a pattern to all the unallocated space. This eliminates the first
access performance penalty, which may exist on some virtualized
storage (e.g. VMware VMDKs).
* Project accounting and quota #6290 - This features adds project
based usage accounting and quota enforcement to the existing space
accounting and quota functionality. Project quotas add an additional
dimension to traditional user/group quotas. The zfs project and zfs
projectspace subcommands have been added to manage projects, set quota
limits and report on usage.
* Channel programs #6558 - The zpool program subcommand can be used
to perform compound ZFS administrative actions via Lua scripts in a
sandboxed environment (with time and memory limits).
* Pyzfs #7230 - The new pyzfs library is intended to provide a
stable interface for the programmatic administration of ZFS. This
wrapper provides a one-to-one mapping for the libzfs_core API
functions, but the signatures and types are more natural to Python.
* Python 3 compatibility #8096 - The arcstat, arcsummary, and
dbufstat utilities have been updated to be compatible with Python 3.
* Direct IO #7823 - Adds support for Linux's direct IO interface.
Performance
* Sequential scrub and resilver #6256 - When scrubbing or
resilvering a pool the process has been split into two phases. The
first phase scans the pool metadata in order to determine where the
data blocks are stored on disk. This allows the second phase to issue
scrub I/O as sequentially as possible, greatly improving performance.
* Allocation classes #5182 - Allows a pool to include a small number
of high-performance SSD devices that are dedicated to storing specific
types of frequently accessed blocks (e.g. metadata, DDT data, or small
file blocks). A pool can opt-in to this feature by adding a special or
dedup top-level device.
* Administrative commands #7668 - Improved performance due to
targeted caching of the metadata required for administrative commands
like zfs list and zfs get.
* Parallel allocation #7682 - The allocation process has been
parallelized by creating multiple "allocators" per-metaslab group.
This results in improved allocation performance on high-end systems.
* Deferred resilvers #7732 - This feature allows new resilvers to be
postponed if an existing one is already in progress. By waiting for
the running resilver to complete redundancy is restored as quickly as
possible.
* ZFS Intent Log (ZIL) #6566 - New log blocks are created and issued
while there are still outstanding blocks being serviced by the
storage, effectively reducing the overall latency observed by the
application.
* Volumes #8615 - When a pool contains a large number of volumes
they are more promptly registered with the system and made available
for use after a zpool import.
* QAT #7295 #7282 #6767 - Support for accelerated SHA256 checksums,
AES-GCM encryption, and the new QAT Intel(R) C62x Chipset / Atom(R)
C3000 Processor Product Family SoC.
Changes in Behavior
* Relaxed (ref)reservation constraints on volumes, they may now be
set larger than the volume size.
* The arcstat.py, arc_summary.py, and dbufstat.py commands have been
renamed arcstat, arc_summary, and dbufstat respectively.
* The SPL source is now included in the ZFS repository removing the
need for separate packages.
* The dedupditto pool property and zfs send -D option have been
deprecated and will be removed in a future release.
Changes for 0.8.1
* Fix comparison signedness in arc_is_overflowing() #8873
* Fix incorrect error message for raw receive #8863
* arc_summary: prefer python3 version and install when there is no
python #8851
* Fix %post and %postun generation in kmodtool #8866
* Reinstate raw receive check when truncating #8852 #8857
* If $ZFS_BOOTFS contains guid, replace the guid portion with $pool
* Fix integer overflow of ZTOI(zp)->i_generation #8858
* hkdf_test binary should only have one icp instance #8850
* Fixed a small typo in man/man1/raidz_test.1 #8855
* Allow TRIM_UNUSED_KSYM when build as a builtin-module #8820
* Make Python detection optional and more portable #8809 #8731
* Wait in 'S' state when send/recv pipe is blocking #8733 #8752
* Make zfs_async_block_max_blocks handle zero correctly #8829 #8289
* Revert "Report holes when there are only metadata changes" #8816
* Exclude log device ashift from normal class #8735
* Fix integer overflow in get_next_chunk() #8778 #8797
* Double-free of encryption wrapping key due to invalid pool
properties #8791
* Endless loop in zpool_do_remove() on platforms with unsigned char
* Fix embedded bp accounting in count_block() #8800 #8766
* Disable parallel processing for 'zfs mount -l' #8762 #8811
* Linux 5.2 compat: Directly call wait_on_page_bit() #8794
* Linux 5.2 compat: Fix config/kernel-shrink.m4 test failure #8776
* Linux 5.2 compat: Remove config/kernel-set-fs-pwd.m4 #8777
* zpool: status -t is not documented in help message #8782
* VERIFY3P() message is missing a space character #8786
columns #8785
* zfs: don't pretty-print objsetid property #8784
* zfs: missing newline character in zfs_do_channel_program() error
message #8783
* Fix ksh-path for random_readwrite_fixed.ksh #8779
* Linux 2.6.39 compat: Test if kstrtoul() exists #8760 #8761
* Device removal panics on 32-bit systems #8790
* zpool: trim -p is not a valid option #8781
* Fix coverity defects: CID 186143 #8788
* Fix kstat state update during pool transition #8746
* Linux 5.2 compat: rw_tryupgrade() #8730
Changes for 0.8.2
* Disabled resilver_defer feature leads to looping resilvers #9299
#9338
* Fix dsl_scan_ds_clone_swapped logic #9140 #9163
* Scrubbing root pools may deadlock on kernels without
elevator_change() (#9321) #9321
* QAT related bug fixes #9276 #9303
* kmodtool: depmod path #8724 #9310
* Fix /etc/hostid on root pool deadlock #9256 #9285
* BuildRequires libtirpc-devel needed for RHEL 8 #9289
* Fix zpool subcommands error message with some unsupported options
#9270
* Fix zfs-dkms .deb package warning in prerm script #9271
* zvol_wait script should ignore partially received zvols #9260
* New service that waits on zvol links to be created #8975
* Always refuse receving non-resume stream when resume state exists
#9252
* Fix Intel QAT / ZFS compatibility on v4.7.1+ kernels #9268 #9269
* etc/init.d/zfs-functions.in: remove arch warning
* zfs_handle used after being closed/freed in change_one callback
#9165
* Fix zil replay panic when TX_REMOVE followed by TX_CREATE #7151
#8910 #9123 #9145
* zfs_ioc_snapshot: check user-prop permissions on snapshotted
datasets #9179 #9180
* Fix Plymouth passphrase prompt in initramfs script #9202
* Fix deadlock in 'zfs rollback' #9203
* Make slog test setup more robust #9194
* zfs-mount-genrator: dependencies should be space-separated #9174
* Linux 5.3: Fix switch() fall though compiler errors #9170
* Linux 5.3 compat: Makefile subdir-m no longer supported #9169
* Fix out-of-order ZIL txtype lost on hardlinked files #8769 #9061
* Increase default zcmd allocation to 256K #9084
* Improve performance by using dmu_tx_hold_*_by_dnode() #9081
* Fix channel programs on s390x #8992 #9080
* Race between zfs-share and zfs-mount services #9083
* Implement secpolicy_vnode_setid_retain() #9035 #9043
* zed crashes when devid not present #9054 #9060
* Don't directly cast unsigned long to void* #9065
* Fix module_param() type for zfs_read_chunk_size #9051
* Move some tests to cli_user/zpool_status #9057
* Race condition between spa async threads and export #9015 #9044
* hdr_recl calls zthr_wakeup() on destroyed zthr #9047
* Fix wrong comment on zcr_blksz_{min,max} #9052
* Retire unused spl_{mutex,rwlock}_{init_fini} #9029
* Linux 5.3 compat: retire rw_tryupgrade() #9029
* Linux 5.3 compat: rw_semaphore owner #9029
* Fix lockdep recursive locking false positive in dbuf_destroy #8984
* Add missing __GFP_HIGHMEM flag to vmalloc #9031
* Use zfsctl_snapshot_hold() wrapper #9039
* Minor style cleanup #9030
* Fix get_special_prop() build failure #9020
* systemd encryption key support #8750 #8848
* Drop redundant POSIX ACL check in zpl_init_acl() #9009
* Export dnode symbols #9027
* Ensure dsl_destroy_head() decrypts objsets #9021
* Disable unused pathname::pn_path* (unneeded in Linux) #9025
* Fixes: #8934 Large kmem_alloc #8934 #9011
* Fix ZTS killed processes detection #9003
* pkg-utils python sitelib for SLES15 #8969
* Fix race in parallel mount's thread dispatching algorithm #8450
#8833 #8878
* Fix dracut Debian/Ubuntu packaging #8990 #8991
* Remove VERIFY from dsl_dataset_crypt_stats() #8976
* Improve "Unable to automount" error message. #8959
* Check b_freeze_cksum under ZFS_DEBUG_MODIFY conditional #8979
* Fix error text for EINVAL in zfs_receive_one() #8977
* Don't use d_path() for automount mount point for chroot'd process
#8903 #8966
* nopwrites on dmu_sync-ed blocks can result in a panic #8957
* Avoid extra taskq_dispatch() calls by DMU #8909
* -Y option for zdb is valid #8926
* Fix error message on promoting encrypted dataset #8905 #8935
* Fix out-of-tree build failures #8921 #8943
* dn_struct_rwlock can not be held in dmu_tx_try_assign() #8929
* Remove arch and relax version dependency #8914
* Add libnvpair to libzfs pkg-config #8919
* Let zfs mount all tolerate in-progress mounts #8881
* zstreamdump: add per-record-type counters and an overhead counter
#8432
* Fix comments on zfs_bookmark_phys #8945
* Add SCSI_PASSTHROUGH to zvols to enable UNMAP support #8933
* Prevent pointer to an out-of-scope local variable #8924 #8940
* dedup=verify doesn't clear the blkptr's dedup flag #8936
* Update vdev_ops_t from illumos #8925
* Allow unencrypted children of encrypted datasets #8737 #8870
* Replace whereis with type in zfs-lib.sh #8920 #8938
* Use ZFS_DEV macro instead of literals #8912
* Fix memory leak in check_disk() #8897 #8911
* kmod-zfs-devel rpm should provide kmod-spl-devel #8930
* ZTS: Fix mmp_interval failure #8906
* Minimize aggsum_compare(&arc_size, arc_c) calls. #8901
* Python config cleanup #8895
* lz4_decompress_abd declared but not defined #8894
* panic in removal_remap test on 4K devices #8893
* compress metadata in later sync passes #8892
* Move write aggregation memory copy out of vq_lock #8890
* Restrict filesystem creation if name referred either '.' or '..'
#8842 #8564
* ztest: dmu_tx_assign() gets ENOSPC in spa_vdev_remove_thread()
#8889
* Fix lockdep warning on insmod #8868 #8884
* fat zap should prefetch when iterating #8862
* Target ARC size can get reduced to arc_c_min #8864
* Fix typo in vdev_raidz_math.c #8875 #8880
* Improve ZTS block_device_wait debugging #8839
* Block_device_wait does not return an error code #8839
* Remove redundant redundant remove #8839
* Fix logic error in setpartition function #8839
* Allow metaslab to be unloaded even when not freed from #8837
* Avoid updating zfs_gitrev.h when rev is unchanged #8860
* l2arc_apply_transforms: Fix typo in comment #8822
* Reduced IOPS when all vdevs are in the
zfs_mg_fragmentation_threshold #8859
* Drop objid argument in zfs_znode_alloc() (sync with OpenZFS) #8841
* Remove vn_set_fs_pwd()/vn_set_pwd() (no need to be at / during
insmod) #8826
* grammar: it is / plural agreement #8818
* Refactor parent dataset handling in libzfs zfs_rename() #8815
* Update comments to match code #8759
* Update descriptions for vnops #8767
* Drop local definition of MOUNT_BUSY #8765
* kernel timer API rework #8647
Changes for 0.8.3
* Fix zfs-0.8.3 "qat.h"
* Prevent unnecessary resilver restarts #9155 #9378 #9551 #9588
* Fix QAT allocation failure return value #9784 #9788
* Fix zfs-0.8.3 zfs_receive_raw test case
* zdb: print block checksums with 6 d's of verbosity
* zfs-load-key.sh: ${ZFS} is not the zfs binary #9780
* Avoid some crashes when importing a pool with corrupt metadata #9022
* In initramfs, do not prompt if keylocation is "file://" #9764
* libspl: declare aok extern in header #9752
* Cancel initialize and TRIM before vdev_metaslab_fini() #8602 #9751
* Update maximum kernel version to 5.4 #9754 #9759
* Fix for ARC sysctls ignored at runtime
* cppcheck: (warning) Possible null pointer dereference: nvh #9732
* cppcheck: (error) Address of local auto-variable assigned #9732
* cppcheck: (error) Null pointer dereference: who_perm #9732
* cppcheck: (warning) Possible null pointer dereference: dnp #9732
* cppcheck: (error) Memory leak: vtoc #9732
* cppcheck: (error) Shifting signed 64-bit value by 63 bits #9732
* cppcheck: (error) Uninitialized variable #9732
* Exchanged two "${ZFS} get -H -o value" commands #9736
* Create symbolic links in /dev/disk/by-vdev for nvme disk devices
#9730
* Force systems with kernel option "quiet" to display prompt for
password #9731
* initramfs: setup keymapping and video for prompts #9723
* Don't fail to apply umask for O_TMPFILE files #8997 #8998
* Allow empty ds_props_obj to be destroyed #9704
* Fix use-after-free of vd_path in spa_vdev_remove() #9706
* zio_decompress_data always ASSERTs successful decompression #9612
#9630
* Exclude data from cores unconditionally and metadata conditionally
#9691
* Set send_realloc_files.ksh to use properties.shlib #9679
* Fix reporting of L2ARC hits/misses in arc_summary3 #9669
* Fix zdb_read_block using zio after it is destroyed #9644 #9657
* Fix use-after-free in case of L2ARC prefetch failure #9648
* Increase allowed 'special_small_blocks' maximum value #9131 #9355
* Adapt gitignore for modules #9656
* Fix encryption logic in systemd mount generator #9611
* Fix non-absolute path in systemd mount generator #9611
* Fix small typo in systemd mount generator #9611
* Implement -A (ignore ASSERTs) for zdb #9610
* Remove zfs_vdev_elevator module option #9417 #9609
* Add display of checksums to zdb -R #9607
* Check for unlinked znodes after igrab() #9602
* Remove requirement for -d 1 for zfs list and zfs get with bookmarks
#9589
* Break out of zfs_zget early if unlinked znode #9583
* Remove inappropiate error message suggesting to use '-r' #9574
* Change zed.service to zfs-zed.service in man page #9581
* Prevent NULL pointer dereference in blkg_tryget() on EL8 kernels
#9546 #9577
* Add missing documentation for some KMC flags #9034
* Fix zpool create -o <property> error message #9550 #9568
* Improve logging of 128KB writes #9409
* Skip loading already loaded key #9495 #9529
* Add a notice in /etc/defaults/zfs for systemd users #9544
* Include prototypes for vdev_initialize #9535
* dracut/zfs-load-key.sh: properly remove prefixes #9520
* Fix contrib/zcp/Makefile.am #9527
* Fix 'zfs change-key' with unencrypted child #9524
* Fix zpool history unbounded memory usage #9516
* Fix incremental recursive encrypted receive #9494
* Use correct format string when printing int8 #9486
* Name anonymous enum of KMC_BIT constants #9478
* Update skc_obj_alloc for spl kmem caches that are backed by Linux
#9474
* Modify sharenfs=on default behavior #9397 #9425
* Implement ZPOOL_IMPORT_UDEV_TIMEOUT_MS #9436
* Clarify loop variable name in zfs copies test #9445
* Fix pool creation with feature@allocation_classes disabled #9427
#9429
* Update zfs program command usage #9056 #9428
* Fix automount for root filesystems #9381 #9384
* Rename rangelock_ functions to zfs_rangelock_ #9402
* Workaround to avoid a race when /var/lib is a persistent dataset
#9360
* Fix for zfs-dracut regression #8913 #9379
* Perform KABI checks in parallel #8547 #9132 #9341
* SIMD: Use alloc_pages_node to force alignment #9608 #9674
* Linux 5.0 compat: SIMD compatibility
* Add warning for zfs_vdev_elevator option removal #9317
* diff_cb() does not handle large dnodes #7678 #8931 #9343
* Use signed types to prevent subtraction overflow #9355
* Refactor libzfs_error_init newlines #9330
* Device removal of indirect vdev panics the kernel #9327
* Fix clone handling with encryption roots #9267 #9294
* Canonicalize Python shebangs #9314
* Fix stalled txg with repeated noop scans #9300
* Clean up do_vol_test in zfs_copies tests #9286
* Fix noop receive of raw send stream #9221 #9173
* Clean up zfs_clone_010_pos #9284
* Refactor checksum operations in tests #9280
* Use the right booleans #9264
* Fix panic on DilOS with kstat per dataset statistics #9254 #9151
* maxinflight can overflow in spa_load_verify_cb() #9272
* Fix typos #9251
* Fix typos in module/zfs/ #9240
* Fix typos in lib/ #9237
* Fix typos in module/ #9241
* Fix typos in modules/icp/ #9239
* Fix typos in include/ #9238
* Fix typos in etc/ #9236
* Fix typos in contrib/ #9235
* Fix typos in cmd/ #9234
* Fix typos in man/ #9233
* Fix typos in config/ #9232
* Fix refquota_007_neg.ksh #9257
* Prevent metaslab_sync panic due to spa_final_dirty_txg #9185 #9186
#9231 #9253
* Simplify deleting partitions in libtest #9224
* Use compatible arg order in tests #9228
* Use smaller default slack/delta value for schedule_hrtimeout_range()
#9217
* Prefer for(;;) to while (TRUE) #9219
* Add regression test for "zpool list -p" #9134
* Split argument list, satisfy shellcheck SC2086 #9212
* Fix install error introduced by #9089
* Document ZFS_DKMS_ENABLE_DEBUGINFO in userland configuration #9191
* Dedup IOC enum values in libzfs_input_check #9188
* Enhance ioctl number checks #9187
* Minor cleanup in Makefile.am #9189
* zfs-functions.in: in_mtab() always returns 1 #9168
* Fix lockdep circular locking false positive involving sa_lock #9110
* Set "none" scheduler if available (initramfs) #9042
* Add more refquota tests #9139
* initramfs: fixes for (debian) initramfs #7904 #9089
* dmu_tx_wait() hang likely due to cv_signal() in
dsl_pool_dirty_delta() #9137
* Improve write performance by using dmu_read_by_dnode() #9156
* Assert that a dnode's bonuslen never exceeds its recorded size #8348
* Make txg_wait_synced conditional in zfsvfs_teardown #9115
* Prevent race in blkptr_verify against device removal #9112
* Fix device expansion when VM is powered off #9111
* spa_load_verify() may consume too much memory #9146
* Change boolean-like uint8_t fields in znode_t to boolean_t #9092
* Drop KMC_NOEMERGENCY #9119
* Don't wakeup unnecessarily in 'zpool events -f' #9091
* Test cancelling a removal in ZTS #9101
* lockdep false positive - move txg_kick() outside of ->dp_lock #9094
* Add channel program for property based snapshots #8443 #9050
* install path fixes #9087
* Don't activate metaslabs with weight 0 #8968
* OpenZFS 9318 - vol_volsize_to_reservation does not account for raidz
skip blocks #8973
* Concurrent small allocation defeats large allocation #8843
* Fix bp_embedded_type enum definition #8951
* OpenZFS 9425 - channel programs can be interrupted #8904
* looping in metaslab_block_picker impacts performance on fragmented
pools #8877
* single-chunk scatter ABDs can be treated as linear #8580
* make zil max block size tunable #8865
Test-Parameters: testlist=sanity,sanity-hsm,sanityn,sanity-quota fstype=zfs ostcount=2 mdscount=2
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I28e9cd70e56a2d73fc9b8347a9ddfe28e0a85090
Reviewed-on: https://review.whamcloud.com/37373
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
git://git.whamcloud.com / fs / lustre-release.git / commitdiff