Whamcloud - gitweb
tools/e2fsprogs.git
8 months agoMerge tag 'v1.45.6' into next master
Theodore Ts'o [Sun, 22 Mar 2020 02:34:30 +0000 (22:34 -0400)]
Merge tag 'v1.45.6' into next

v1.45.6

8 months agoUpdate release notes, etc., for the 1.45.6 release
Theodore Ts'o [Sat, 21 Mar 2020 03:51:07 +0000 (23:51 -0400)]
Update release notes, etc., for the 1.45.6 release

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoAOSP: tune2fs, resize2fs: make ramdisk binaries.
Jaegeuk Kim [Wed, 12 Feb 2020 19:34:49 +0000 (11:34 -0800)]
AOSP: tune2fs, resize2fs: make ramdisk binaries.

Bug: 149391799
Change-Id: I5183755915710e28a603e3f718f16813ea9991a0
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
From AOSP commit: a13a88d0d557a396f63702fb8db008487e2384d7

8 months agoAOSP: e2fsdroid: Don't skip unusable blocks in BaseFS.
David Anderson [Fri, 14 Feb 2020 20:44:48 +0000 (12:44 -0800)]
AOSP: e2fsdroid: Don't skip unusable blocks in BaseFS.

Currently, basefs_allocator will iterate through blocks owned by an
inode until it finds a block that is free. This effectively ignores the
logical to physical block mapping, which can lead to a bigger delta in
the final image.

An example of how this can happen is if the BaseFS has a deduplicated
block (D), that is not deduplicated in the new image:

  Old image: 1 2 3 D 4 5
  New image: 1 2 3 ? 4 5

If the allocator sees that "D" is not usable, and skips to block "4",
we will have a non-ideal assignment.

  Bad image: 1 2 3 4 5 ?

This patch refactors get_next_block() to acquire at most one block. It's
called a single time, and then only called in a loop if absolutely no
blocks can be acquired from anywhere else.

In a Virtual A/B simulation, this reduces the COW snapshot size by about
90MB.

Bug: 139201772
Test: manual test
Change-Id: I354f0dee1ee191dba0e1f90491ed591dba388f7f
From AOSP commit: a495b54f89b2ec0e46be8e3564e4852c6434687c

8 months agoAOSP: e2fsdroid: Fix logical block sequencing in BaseFS.
David Anderson [Fri, 14 Feb 2020 03:20:32 +0000 (19:20 -0800)]
AOSP: e2fsdroid: Fix logical block sequencing in BaseFS.

By iterating over blocks to write BaseFS, holes in the extent tree are
skipped. This is problematic because the purpose of BaseFS is to
preserve the logical to physical block assignment between builds. By not
preserving the location of holes, the assignment can be incorrect.

For example, consider the following block list for a file:
   1 2 3 0 4 5

If this is recorded as:
   1 2 3 4 5

If the first block changes to a hole, the intended mapping will not be
preserved at all:
   0 1 2 0 3

This patch makes two changes to e2fsdroid to fix this. The first change
is that holes are now recorded in BaseFS, by iterating over the extent
tree rather than the block list, and inserting zeroes where appropriate.

The second change is that the block allocator now recognizes when blocks
have been skipped (either to deduplication or to holes), and skips the
same number of logical blocks in BaseFS as well.

In a Virtual A/B simulation, this reduces the COW snapshot size by
approximately 100MB.

Bug: 139201772
Test: m target-files-package, inspect .map files
From AOSP commit: d391f3bf38cbe51718d5c3c0bb3e72b1a9978625

8 months agoAOSP: e2fsdroid: Properly free the dedup block map.
David Anderson [Wed, 29 Jan 2020 23:31:14 +0000 (15:31 -0800)]
AOSP: e2fsdroid: Properly free the dedup block map.

When BaseFS specifies the same block for two files, it gets added to a
separate "dedup" bitmap, and removed from the free block bitmap. If the
new build does not use every block in this bitmap, there will be an
inconsistency: the block bitmap marks blocks as in-use when they are
actually free. Although this doesn't matter for AOSP's read-only file
systems, it does cause e2fsck to complain, which breaks the build.

Fix the inconsistency by properly freeing all unused blocks within the
dedup block set.

Bug: 139201772
Test: build AOSP using BaseFS
Change-Id: I6b6511eb713a56fec932f1d5668f1766d64d9479
From AOSP commit: 346bee6f8b97aefe7714688f738606c116099fbc

8 months agoAOSP: Build e2freefrag
Alessio Balsini [Fri, 24 Jan 2020 16:04:44 +0000 (16:04 +0000)]
AOSP: Build e2freefrag

Enable the build of e2freefrag to monitor the free space fragmentation
in ext2/3/4 filesystems.

Bug: 146078546
Test: m + e2freefrag on device
Change-Id: Ia56e443a789ae881a03bdaeae1093567e1736c4c
Signed-off-by: Alessio Balsini <balsini@google.com>
From AOSP commit: ab77f6c79f3dab697cd56ad3b793e7d555ac9415

8 months agoAOSP: Add -e2fsprogs to the e2fsprogs chattr and lsattr.
Elliott Hughes [Thu, 23 Jan 2020 23:44:10 +0000 (15:44 -0800)]
AOSP: Add -e2fsprogs to the e2fsprogs chattr and lsattr.

We want to start shipping the toybox chattr and lsattr on the device all
the time, so the build system rightly complains that then we'd have two
modules with the same name.

I went with a suffix rather than a prefix so that tab completion works
for folks still wanting to use the e2fsprogs versions.

Bug: http://b/147769529
Test: builds
Change-Id: Ib904fa6c709d29ce709302c61e452383c02cb9a3
From AOSP commit: 8525a455e7410461560a99a42feb0dbfabab5c8e

8 months agoAOSP: Make ramdisk_available.
Yifan Hong [Wed, 22 Jan 2020 00:44:20 +0000 (16:44 -0800)]
AOSP: Make ramdisk_available.

Test: pass
Bug: 147347110
Change-Id: Ie800ba1b56773dcc1b6563c4f19c27eccb9ffc1a
From AOSP commit: f5a8e8fdefd78deae971a475a7fa43734eef205e

8 months agoAOSP: Change #define to _BLKID_TYPES_H
Kousik Kumar [Fri, 10 Jan 2020 00:15:30 +0000 (16:15 -0800)]
AOSP: Change #define to _BLKID_TYPES_H

blkid_types.h and ext_types.h having the exact same content results in
mismatches in remote RBE builds. Given blkid_types.h is actually
supposed to be different, changing this to remove the mismatch.

Test: Ran a build, and all e2fsprogs mismatches went away between
local/remote.

Change-Id: I63ab1719ee1d0ccd28907f0bc99531260251fa99
From AOSP commit: ec10b513c283706f984edeec47301b0661f7d283

8 months agoAOSP: Allow resize2fs to compile with BUILD_HOST_static
Dario Freni [Mon, 6 Jan 2020 15:57:14 +0000 (15:57 +0000)]
AOSP: Allow resize2fs to compile with BUILD_HOST_static

Bug: 144477678
Test: BUILD_HOST_static=1 m resize2fs; m resize2fs
Change-Id: I0986deccfe496153e662dcc3cc2fb1ffb6973c21
From AOSP commit: 2c767b86591c64cd7b84c5619e8d8b8a0afd557e

8 months agoAOSP: Allow debugfs_static to be compiled as host tool.
Dario Freni [Thu, 2 Jan 2020 12:59:39 +0000 (12:59 +0000)]
AOSP: Allow debugfs_static to be compiled as host tool.

Bug: 144477678
Test: m debugfs_static
Change-Id: I7c360a2a381f8508578d14c32bbb280f386dd925
From AOSP commit: 742bb05a401eb2feb6caaee1c8d66fc1c37eef77

8 months agopo: update ms.po (from translationproject.org)
Sharuzzaman Ahmat Raslan [Sat, 21 Mar 2020 02:31:25 +0000 (22:31 -0400)]
po: update ms.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agomisc: add text describing the impact of an inode size < 128 bytes in man pages
Theodore Ts'o [Sat, 21 Mar 2020 01:35:03 +0000 (21:35 -0400)]
misc: add text describing the impact of an inode size < 128 bytes in man pages

Addresses-Debian-Bug: #953493
Addresses-Debian-Bug: #953494
Addresses-Debian-Bug: #951808
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agolibe2p: add a thread-safe variant of e2p_feature2string
Theodore Ts'o [Fri, 20 Mar 2020 21:57:38 +0000 (17:57 -0400)]
libe2p: add a thread-safe variant of e2p_feature2string

This commit adds the function e2p_feature_to_string() which allows the
caller to pass in a preallocated buffer.

Google-Bug-Id: 16978603
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agolibext2fs: fix the {set_get}_bitmap_range functions when bitmap->start > 7
Theodore Ts'o [Fri, 20 Mar 2020 19:24:18 +0000 (15:24 -0400)]
libext2fs: fix the {set_get}_bitmap_range functions when bitmap->start > 7

The bitmap array's set/get bitmap_range functions were not subtracting
out bitmap->start.  This doesn't matter for normal file systems, since
the bitmap->start is zero or one, and the passed-in starting range is
a multiple of eight, and the starting range is then divided by 8.

But with a non-standard/fuzzed file system, bitmap->start could be
significantly larger, and this could then lead to a array out of
bounds memory reference.

Google-Bug-Id: 147849134
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: clarify overflow link count error message
Jan Kara [Thu, 13 Feb 2020 10:15:56 +0000 (11:15 +0100)]
e2fsck: clarify overflow link count error message

When directory link count is set to overflow value (1) but during pass 4
we find out the exact link count would fit, we either silently fix this
(which is not great because e2fsck then reports the fs was modified but
output doesn't indicate why in any way), or we report that link count is
wrong and ask whether we should fix it (in case -n option was
specified). The second case is even more misleading because it suggests
non-trivial fs corruption which then gets silently fixed on the next
run. Similarly to how we fix up other non-problems, just create a new
error message for the case directory link count is not overflown anymore
and always report it to clarify what is going on.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
(cherry picked from commit 4ebce13292f54c96f43dcb1bd1d5b8df5dc8749d)

8 months agoMerge branch 'maint' into next
Theodore Ts'o [Mon, 16 Mar 2020 11:59:08 +0000 (07:59 -0400)]
Merge branch 'maint' into next

8 months agotune2fs: update dir checksums when clearing dir_index feature
Jan Kara [Thu, 13 Feb 2020 10:16:02 +0000 (11:16 +0100)]
tune2fs: update dir checksums when clearing dir_index feature

When clearing dir_index feature while metadata_csum is enabled, we have
to rewrite checksums of all indexed directories to update checksums of
internal tree nodes.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agotests: add test to excercise indexed directories with metadata_csum
Jan Kara [Thu, 13 Feb 2020 10:16:01 +0000 (11:16 +0100)]
tests: add test to excercise indexed directories with metadata_csum

Indexed directories have somewhat different format when metadata_csum is
enabled. Add test to excercise linking in indexed directories and e2fsck
rehash code in this case.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agotests: modify f_large_dir test to excercise indexed dir handling
Jan Kara [Thu, 13 Feb 2020 10:16:00 +0000 (11:16 +0100)]
tests: modify f_large_dir test to excercise indexed dir handling

Modify f_large_dir test to create indexed directory and create entries
in it. That way the new code in ext2fs_link() for addition of entries
into indexed directories gets executed including various special cases
when growing htree.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoext2fs: implement dir entry creation in htree directories
Jan Kara [Thu, 13 Feb 2020 10:15:59 +0000 (11:15 +0100)]
ext2fs: implement dir entry creation in htree directories

Implement proper creation of new directory entries in htree directories
in ext2fs_link(). So far we just cleared EXT2_INDEX_FL and treated
directory as unindexed however this results in mismatched checksums if
metadata checksums are in use because checksums are placed in different
places depending on htree node type.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoext2fs: update allocation info earlier in ext2fs_mkdir() and ext2fs_symlink()
Jan Kara [Thu, 13 Feb 2020 10:15:58 +0000 (11:15 +0100)]
ext2fs: update allocation info earlier in ext2fs_mkdir() and ext2fs_symlink()

Currently, ext2fs_mkdir() and ext2fs_symlink() update allocation bitmaps
and other information only close to the end of the function, in
particular after calling to ext2fs_link(). When ext2fs_link() will
support indexed directories, it will also need to allocate blocks and
that would cause filesystem corruption in case allocation info isn't
properly updated. So make sure ext2fs_mkdir() and ext2fs_symlink()
update allocation info before calling into ext2fs_link().

[ Added error handling so the calls to ext2fs_{block,inode}_alloc_stats()
  can be undone if the newly created directory or symlink can not be linked
  into the directory. -- TYT ]

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agodebian: drop libattr1-dev from the build dependencies list
Theodore Ts'o [Sun, 15 Mar 2020 04:56:01 +0000 (00:56 -0400)]
debian: drop libattr1-dev from the build dependencies list

The libattr has stopped providing attr/xattr.h; we now use
sys/xattr.h.  So there is no longer any reason to require that the
libattr1-dev package be present when building e2fsprogs, so drop it.

Addresses-Debian-Bug: #953926
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: fix "make check" when using static libraries
Theodore Ts'o [Sun, 15 Mar 2020 04:26:22 +0000 (00:26 -0400)]
e2fsck: fix "make check" when using static libraries

Fixes: 70303df16ca6 ("e2fsck: consistently use ext2fs_get_mem()")

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agolibext2fs: make ext2fs_dirent_has_tail() more strict
Theodore Ts'o [Sun, 15 Mar 2020 03:24:39 +0000 (23:24 -0400)]
libext2fs: make ext2fs_dirent_has_tail() more strict

Previously ext2fs_dirent_has_tail() would return true if the directory
was corrupted.  If the directory is corrupted, then by definition it
doesn't have a valid checksum tail.

(This fixes a big-endian failure on the master branch.)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agomisc: fix typos in chattr's man page
Sawood Alam [Tue, 10 Mar 2020 02:47:29 +0000 (22:47 -0400)]
misc: fix typos in chattr's man page

Plural form "directories" should be used along with "files".
"id's" should be "ids" (i.e., plural form, not apostrophe).
"much" should "must".

Signed-off-by: Sawood Alam <ibnesayeed@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agomke2fs.conf: remove options.fname_encoding
Pino Toscano [Mon, 9 Mar 2020 16:36:10 +0000 (12:36 -0400)]
mke2fs.conf: remove options.fname_encoding

Introduced with commit e7236a9476cd1fa5296fbc4aa573b36426901a08,
it was later renamed to encoding, and turned into a fs_type-only
option with commit 28887533bb64db318e74c38cd9c0ad6d0bb2ced2.

Hence, remove an option that does not exist in the default
configuration.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: fix indexed dir rehash failure with metadata_csum enabled
Jan Kara [Thu, 13 Feb 2020 10:15:57 +0000 (11:15 +0100)]
e2fsck: fix indexed dir rehash failure with metadata_csum enabled

E2fsck directory rehashing code can fail with ENOSPC due to a bug in
ext2fs_htree_intnode_maxrecs() which fails to take metadata checksum
into account and thus e.g. e2fsck can decide to create 1 indirect level
of index tree when two are actually needed. Fix the logic to account for
metadata checksum.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: clarify overflow link count error message
Jan Kara [Thu, 13 Feb 2020 10:15:56 +0000 (11:15 +0100)]
e2fsck: clarify overflow link count error message

When directory link count is set to overflow value (1) but during pass 4
we find out the exact link count would fit, we either silently fix this
(which is not great because e2fsck then reports the fs was modified but
output doesn't indicate why in any way), or we report that link count is
wrong and ask whether we should fix it (in case -n option was
specified). The second case is even more misleading because it suggests
non-trivial fs corruption which then gets silently fixed on the next
run. Similarly to how we fix up other non-problems, just create a new
error message for the case directory link count is not overflown anymore
and always report it to clarify what is going on.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoMerge branch 'maint' into next
Theodore Ts'o [Sat, 7 Mar 2020 18:13:55 +0000 (13:13 -0500)]
Merge branch 'maint' into next

8 months agolibext2fs: check open(O_EXCL) first in ismounted.c
Lukas Czerner [Tue, 3 Mar 2020 13:53:48 +0000 (14:53 +0100)]
libext2fs: check open(O_EXCL) first in ismounted.c

Currently the ext2fs_check_mount_point() will use the open(O_EXCL) check
on linux after all the other checks are done. However it is not
necessary to check mntent if open(O_EXCL) succeeds because it means that
the device is not mounted.

Moreover the commit ea4d53b7 introduced a regression where a following
set of commands fails:

vgcreate mygroup /dev/sda
lvcreate -L 1G -n lvol0 mygroup
mkfs.ext4 /dev/mygroup/lvol0
mount /dev/mygroup/lvol0 /mnt
lvrename /dev/mygroup/lvol0 /dev/mygroup/lvol1
lvcreate -L 1G -n lvol0 mygroup
mkfs.ext4 /dev/mygroup/lvol0   <<<--- This fails

It fails because it thinks that /dev/mygroup/lvol0 is mounted because
the device name in /proc/mounts is not updated following the lvrename.

Move the open(O_EXCL) check before the mntent check and return
immediatelly if the device is not busy.

Fixes: ea4d53b7 ("libext2fs/ismounted.c: check device id in advance to skip false device names")
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
Reported-by: Karel Zak <kzak@redhat.com>
Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agomke2fs: fix permissions setting with "mke2fs -d /path/files"
Theodore Ts'o [Sat, 7 Mar 2020 17:35:48 +0000 (12:35 -0500)]
mke2fs: fix permissions setting with "mke2fs -d /path/files"

Set the directory for directories in cases where the owner permissions
is not rwx.  This was reported[1] by Robert Yang but we are using a
different approach to fixing the issue.

[1] https://lore.kernel.org/r/1582542522-97508-1-git-send-email-liezhi.yang@windriver.com

Also set the permissions in a more portable way by making a
distinction between the host OS's permissions stats and Linux's
permissions.  We still assume the low 12 bits are the historical Unix
assignments, but we don't assume ST_IFMT bits are the same as Linux's.

Reported-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agolibext2fs: don't use O_DIRECT for files on tmpfs
Andreas Dilger [Fri, 21 Feb 2020 21:40:56 +0000 (14:40 -0700)]
libext2fs: don't use O_DIRECT for files on tmpfs

If a filesystem image is on tmpfs, opening it with O_DIRECT for
reading the MMP will fail.  This is unnecessary, since the image
file can't really be open on another node at this point.  If the
open with O_DIRECT fails, retry without it when plausible.

Remove the special-casing of tmpfs from the mmp test cases.

Change-Id: I41f4b31657b06f62f10be8d6e524d303dd36a321
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: avoid overflow with very large dirs
Andreas Dilger [Wed, 12 Feb 2020 01:07:21 +0000 (18:07 -0700)]
e2fsck: avoid overflow with very large dirs

In alloc_size_dir() it multiples signed ints when allocating the
buffer for rehashing an htree-indexed directory.  This will overflow
when the directory size is above 4GB, which is possible with largedir
directories having about 100M entries, assuming an average 3/4 leaf
fullness and 24-byte filenames, or fewer with longer filenames.
The same problem exisgs in get_next_block().

Similarly, the out_dir struct used a signed int for the number of
blocks in the directory, which may result in a negative size if the
directory is over 2GB (about 50M entries or fewer).

Use appropriate unsigned variables for block counts, and use larger
types for calculating the byte count for memory offsets/sizes.

Such large directories not been seen yet, but are not too far away.
The ext2fs_get_array() function will properly calculate the needed
memory allocation, and detect overflow on 32-bit systems.
Add ext2fs_resize_array() to do the same for array resize.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agomisc: handle very large files with filefrag
Andreas Dilger [Fri, 7 Feb 2020 01:09:46 +0000 (18:09 -0700)]
misc: handle very large files with filefrag

Avoid overflowing the column-width calc printing files over 4B blocks.

Document the [KMG] suffixes for the "-b <blocksize>" option.

The blocksize is limited to at most 1GiB blocksize to avoid shifting
all extents down to zero GB in size.  Even the use of 1GB blocksize
is unlikely, but non-ext4 filesystems may use multi-GB extents.

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
8 months agoe2fsck: consistently use ext2fs_get_mem()
Andreas Dilger [Fri, 7 Feb 2020 01:09:45 +0000 (18:09 -0700)]
e2fsck: consistently use ext2fs_get_mem()

Consistently use ext2fs_get_mem() and ext2fs_free_mem() instead of
calling malloc() and free() directly in e2fsck.  In several places
it is possible to use ext2fs_get_memzero() instead of explicitly
calling memset() on the memory afterward.

This is just a code cleanup, and does not fix any specific bugs.

[ Fix up library dependencies in e2fsck/Makefile.in to fix "make
  check" breakages. -- TYT ]

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agoe2fsck: fix overflow if more than 4B inodes
Andreas Dilger [Fri, 7 Feb 2020 01:09:44 +0000 (18:09 -0700)]
e2fsck: fix overflow if more than 4B inodes

Even though we don't have support for filesystems with over 4B inodes
in the current e2fsprogs, this may happen in the future.  There are
latent overflow bugs when calculating the number of inodes in the
filesystem that can trivially be fixed now, rather than waiting for
them to be hit at some point in the future.  The block number calcs
are already correct in this code.

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agodebugfs: print inode numbers as unsigned
Andreas Dilger [Fri, 7 Feb 2020 01:09:43 +0000 (18:09 -0700)]
debugfs: print inode numbers as unsigned

Print inode numbers as unsigned values, to avoid printing negative
numbers for inodes above 2B.

Flags should be printed as hex instead of signed decimal values.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agodebugfs: allow comment lines in command file
Andreas Dilger [Fri, 7 Feb 2020 01:09:42 +0000 (18:09 -0700)]
debugfs: allow comment lines in command file

Allow comment lines with '#' at the start of the line in the command
file passed in to debugfs via the "-f" option or from standard input.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agoe2fsck: reduce memory usage for many directories
Andreas Dilger [Fri, 7 Feb 2020 01:09:41 +0000 (18:09 -0700)]
e2fsck: reduce memory usage for many directories

Pack struct dx_dir_info and dx_dirblock_info properly in memory, to
avoid holes, and fields are not larger than necessary.  This reduces
the memory needed for each hashed dir, according to pahole(1) from:

    struct dx_dir_info {
        /* size: 32, cachelines: 1, members: 6 */
        /* sum members: 26, holes: 1, sum holes: 2 */
        /* padding: 4 */
    };
    struct dx_dirblock_info {
        /* size: 56, cachelines: 1, members: 9 */
        /* sum members: 48, holes: 2, sum holes: 8 */
        /* last cacheline: 56 bytes */
    };

to 8 bytes less for each directory and directory block, and leaves
space for future use if needed (e.g. larger numblocks):

    struct dx_dir_info {
        /* size: 24, cachelines: 1, members: 6 */
        /* sum members: 20, holes: 1, sum holes: 4 */
        /* bit holes: 1, sum bit holes: 7 bits */
    };
    struct dx_dirblock_info {
        /* size: 48, cachelines: 1, members: 9 */
    };

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agoe2fsck: avoid mallinfo() if over 2GB allocated
Andreas Dilger [Fri, 7 Feb 2020 01:09:40 +0000 (18:09 -0700)]
e2fsck: avoid mallinfo() if over 2GB allocated

Don't use mallinfo() for determining the amount of memory used if it
is over 2GB.  Otherwise, the signed ints used by this interface can
can overflow and return garbage values.  This makes the actual amount
of memory used by e2fsck misleading and hard to determine.

Instead, use brk() to get the total amount of memory allocated, and print
this if the more detailed mallinfo() information is not suitable for use.

There does not appear to be a mallinfo64() variant of this function.
There does appear to be an abomination named malloc_info() that writes
XML-formatted malloc stats to a FILE stream that would need to be read
and parsed in order to get these stats, but that doesn't seem worthwhile.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agoe2fsck: use proper types for variables
Andreas Dilger [Fri, 7 Feb 2020 01:09:39 +0000 (18:09 -0700)]
e2fsck: use proper types for variables

Use ext2_ino_t instead of ino_t for referencing inode numbers.
Use loff_t for for file offsets, and dgrp_t for group numbers.

Cast products to ssize_t before multiplication to avoid overflow.

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agoe2fsck: fix e2fsck_allocate_memory() overflow
Andreas Dilger [Fri, 7 Feb 2020 01:09:38 +0000 (18:09 -0700)]
e2fsck: fix e2fsck_allocate_memory() overflow

e2fsck_allocate_memory() takes an "unsigned int size" argument, which
will overflow for allocations above 4GB.  This happens for dir_info
and dx_dir_info arrays when there are more than 350M directories in a
filesystem, and for the dblist array above 180M directories.

There is also a risk of overflow during the binary search in both
e2fsck_get_dir_info() and e2fsck_get_dx_dir_info() when the midpoint
of the array is calculated, if there would be more than 2B directories
in the filesystem and working above the half way point.

Also, in some places inode numbers are "int" instead of "ext2_ino_t",
which can also cause problems with the array size calculations, and
makes it hard to identify where inode numbers are used.

Fix e2fsck_allocate_memory() to take an "unsigned long" argument to
match ext2fs_get_mem(), so that it can do single memory allocations
over 4GB.

Fix e2fsck_get_dir_info() and e2fsck_get_dx_dir_info() to temporarily
use an unsigned long long value to calculate the midpoint (which will
always fit into an ext2_ino_t again afterward).

Change variables that hold inode numbers to be ext2_ino_t, and print
them as unsigned values instead of printing negative inode numbers.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU-13197
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agotst_libext2fs: Avoid multiple definition of global variables
Lukas Czerner [Mon, 10 Feb 2020 15:24:59 +0000 (16:24 +0100)]
tst_libext2fs: Avoid multiple definition of global variables

gcc version 10 changed the default from -fcommon to -fno-common and as a
result e2fsprogs make check tests fail because tst_libext2fs.c end up
with a build error.

This is because it defines two global variables debug_prog_name and
extra_cmds that are already defined in debugfs/debugfs.c. With -fcommon
linker was able to resolve those into the same object, however with
-fno-common it's no longer able to do it and we end up with multiple
definition errors.

Fix the problem by using SKIP_GLOBDEFS macro to skip the variables
definition in debugfs.c. Note that debug_prog_name is also defined in
lib/ext2fs/extent.c when DEBUG macro is used, but this does not work even
with older gcc versions and is never used regardless so I am not going to
bother with it.

Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agochattr.1: improve attribute formatting with labels and indented paragraphs
Jeremy Visser [Mon, 3 Feb 2020 02:37:41 +0000 (13:37 +1100)]
chattr.1: improve attribute formatting with labels and indented paragraphs

By convention, lists of options in man pages use a label followed by an
indented description, such as this example from the Options section:

     -R     Recursively change attributes of directories and
            their contents.

But the Attributes section places the available attributes mid-sentence,
which makes it visually more difficult to parse:

     A file with the 'a' attribute set can only be opened
     in append mode for writing.  [...]

     When a file with the 'A' attribute set is accessed, its
     atime record is not modified.  [...]

This patch places a label beside each attribute description, which (in
my opinion) improves readability, especially when visually skimming the
list.  For example:

     a      A file with the 'a' attribute set can only be
            opened in append mode for writing.

     A      When a file with the 'A' attribute set is accessed,
            its atime record is not modified.

Signed-off-by: Jeremy Visser <jeremyvisser@google.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agolibext2fs: avoid array buffer overruns caused by insane directory blocks
Theodore Ts'o [Sun, 16 Feb 2020 00:10:01 +0000 (19:10 -0500)]
libext2fs: avoid array buffer overruns caused by insane directory blocks

Reported-by: canardo909@gmx.com
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
9 months agolibext2fs: fix potential OOB read check_for_inode_bad_blocks()
Theodore Ts'o [Thu, 16 Jan 2020 23:14:37 +0000 (18:14 -0500)]
libext2fs: fix potential OOB read check_for_inode_bad_blocks()

If the bad block list has been reset in the middle of an inode scan,
it's possible for bb->list[scan->bad_blocks_ptr] to result in an
out-of-bounds read access.

This is highly unlikely to happen under normal circumstances; in
particular, we generally don't use bad block inodes any more.  In
addition, this would only happen if the bad block inode itself is
corrupt so e2fsck needs to wipe it out.  This might cause e2fsck to
crash, but it will more likely cause a part of the inode table to be
wrongly considered invalid, causing file system to be incorrectly
fixed.

This was reported by TALOS as TALOS-2020-0974 and CVE-2020-6057, but
after closer examination, we don't believe this can be used in any way
to exploit the system or release information about the system, since
all this can do is to cause part of the inode table to be skipped when
it shouldn't be, and this can't be leveraged since any information
about the ASLR of the process is obsolete once e2fsck exits.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agomke2fs: set overhead in super block
Li Dongyang [Mon, 27 Jan 2020 03:06:36 +0000 (22:06 -0500)]
mke2fs: set overhead in super block

If overhead is not recorded in the super block, it is calculated
during mount in kernel, for bigalloc file systems it takes
O(groups**2) in time.  For a 1PB device with 32K cluster size it takes
~12 mins to mount, with most of the time spent on figuring out
overhead.

While we can not improve the overhead algorithm in kernel due to the
nature of bigalloc, we can work out the overhead during mke2fs and set
it in the super block, avoiding calculating it every time when it
mounts.

Overhead is s_first_data_block plus internal journal blocks plus the
block and inode bitmaps, inode table, super block backups and group
descriptor blocks for every group.  This patch introduces
ext2fs_count_used_clusters(), which calculates the clusters used in
the block bitmap for the given range.

When bad blocks are involved, it gets tricky because the blocks
counted as overhead and the bad blocks can end up in the same
allocation cluster.  In this case we will unmark the bad blocks from
the block bitmap, convert to cluster bitmap and get the overhead, then
mark the bad blocks back in the cluster bitmap.

Reset the overhead to zero when resizing, we can not simply count the
used blocks as overhead like we do when mke2fs.  The overhead can be
calculated by kernel side during mount.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoext2fs: rename "s_overhead_blocks" to "s_overhead_clusters"
Li Dongyang [Wed, 20 Nov 2019 04:35:25 +0000 (04:35 +0000)]
ext2fs: rename "s_overhead_blocks" to "s_overhead_clusters"

Rename s_overhead_blocks field from struct ext2_super_block to
make it consistent with the kernel counterpart.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: optimize ext2fs_convert_subcluster_bitmap()
Li Dongyang [Mon, 27 Jan 2020 02:52:02 +0000 (21:52 -0500)]
libext2fs: optimize ext2fs_convert_subcluster_bitmap()

For a bigalloc filesystem, converting the block bitmap from blocks
to chunks in ext2fs_convert_subcluster_bitmap() can take a long time
when the device is huge, because we test the bitmap
bit-by-bit using ext2fs_test_block_bitmap2().
Use ext2fs_find_first_set_block_bitmap2() which is more efficient
for mke2fs when the fs is mostly empty.

e2fsck can also benefit from this during pass1 block scanning.

Time taken for "mke2fs -O bigalloc,extent -C 131072 -b 4096" on a 1PB
device:

without patch:
real    27m49.457s
user    21m36.474s
sys     6m9.514s

with patch:
real    6m31.908s
user    0m1.806s
sys    6m29.697s

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoMerge branch 'maint' into next
Theodore Ts'o [Sat, 25 Jan 2020 08:07:05 +0000 (03:07 -0500)]
Merge branch 'maint' into next

10 months agommp: abstract out repeated 'sizeof(buf), buf' usage
Andreas Dilger [Tue, 14 Jan 2020 21:42:18 +0000 (14:42 -0700)]
mmp: abstract out repeated 'sizeof(buf), buf' usage

The printf("%.*s") format requires both the buffer size and buffer
pointer to be specified for each use.  Since this is repeatedly given
as "(int)sizeof(buf), (char *)buf" for mmp_nodename and mmp_bdevname
fields, with typecasts to avoid compiler warnings.

Add a helper macro EXT2_LEN_STR() to avoid repeated boilerplate code.

This can also be used for other superblock buffer fields that may not
have NUL-terminated strings (e.g. s_volume_name, s_last_mounted,
s_{first,last}_error_func, s_mount_opts) to simplify code and avoid
the need for temporary buffers for NUL-termination.

Annotate the superblock string fields that may not be NUL-terminated.

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agommp: don't assume NUL termination for MMP strings
Andreas Dilger [Tue, 14 Jan 2020 21:42:17 +0000 (14:42 -0700)]
mmp: don't assume NUL termination for MMP strings

Don't assume that mmp_nodename and mmp_bdevname are NUL terminated,
since very long node/device names may completely fill the buffers.

Limit string printing to the maximum buffer size for safety, and
change the field definitions to __u8 to make it more clear that
they are not NUL-terminated strings, as is done with other strings
in the superblock that do not have NUL termination.

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agofilefrag: add -E option to display the extent status cache
Theodore Ts'o [Thu, 23 Jan 2020 15:46:10 +0000 (10:46 -0500)]
filefrag: add -E option to display the extent status cache

Ext4 has an extent status cache; add the fiemap extensions so we can
query the kernel for the extent status cache information.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoe2fsck: restart the full e2fsck run if the bad block inode is invalidate
Theodore Ts'o [Fri, 17 Jan 2020 15:51:16 +0000 (10:51 -0500)]
e2fsck: restart the full e2fsck run if the bad block inode is invalidate

Previously, we just cleared the bad block list and restarted the inode
scan, but we didn't do a full reset of all of e2fsck's state.  When
code handling this case; we didn't have the framework to do a
restarted run.  Now that we do, we can simply the code and make it
more correct.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoe2fsck: clean up unwind_pass1() as it's no longer really needed
Theodore Ts'o [Fri, 17 Jan 2020 02:58:21 +0000 (21:58 -0500)]
e2fsck: clean up unwind_pass1() as it's no longer really needed

We now restart the full e2fsck instead of unwinding and restarting
pass1.  So most of what used to be in unwind_pass1() has been moved
elsewhere.  Let's git rid of it entirely, which simplifies and shrinks
pass1.c slightly.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: don't needlessly byte swap the group descriptors in ext2fs_flush
Theodore Ts'o [Fri, 17 Jan 2020 00:17:26 +0000 (19:17 -0500)]
libext2fs: don't needlessly byte swap the group descriptors in ext2fs_flush

If the EXT2_FLAG_SUPER_ONLY is set, there's no reason to allocate the
shadow block group descriptors and byte swap the group descriptors.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: teach ext2fs_flush() to check if group descriptors are loaded
Theodore Ts'o [Thu, 16 Jan 2020 23:56:49 +0000 (18:56 -0500)]
libext2fs: teach ext2fs_flush() to check if group descriptors are loaded

If the EXT2_FLAG_SUPER_ONLY is not set, and the group descriptors are
not loaded, ext2fs_flush[2]() will return EXT2_ET_NO_GDESC.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoMerge branch 'maint' into next
Theodore Ts'o [Thu, 16 Jan 2020 23:33:24 +0000 (18:33 -0500)]
Merge branch 'maint' into next

10 months agolibext2fs: reserve the error code EXT2_ET_NO_GDESC
Theodore Ts'o [Thu, 16 Jan 2020 20:35:29 +0000 (15:35 -0500)]
libext2fs: reserve the error code EXT2_ET_NO_GDESC

This is really only needed in the 1.46+ where the EXT2_FLAG_SUPER_ONLY
is honored by ext2fs_open to only read the superblock, so that
fs->group_desc can be NULL.  We define it in the maint branch so that
we can be sure the error tables are kept in sync (in the unlikely case
that a new error code needs to be assigned in the maint branch).

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: fix crash in ext2fs_image_super_write() on Big Endian systems
Theodore Ts'o [Tue, 14 Jan 2020 15:58:10 +0000 (10:58 -0500)]
libext2fs: fix crash in ext2fs_image_super_write() on Big Endian systems

This is a similar fix as c9a8c53b17cc ("libext2fs: fix crash in
ext2fs_open2() on Big Endian systems").

Commit e6069a05: ("Teach ext2fs_open2() to honor the
EXT2_FLAG_SUPER_ONLY flag") changed how the function
ext2fs_group_desc() handled a request for a gdp pointer for a group
larger than the number of groups in the file system; it now returns
NULL, instead of returning a pointer beyond the end of the array.

Previously, the ext2fs_imager_super_write() function would swap all of
the block group descriptors in a block, even if they are beyond the
end of the file system.  This was OK, since we were not overrunning
the allocated memory, since it was rounded to a block boundary.  But
now that ext2fs_group_desc() would return NULL for those gdp, it would
cause ext2fs_open2(), when it was byte swapping the block group
descriptors on Big Endian systems, to dereference a null pointer and
crash.

This commit adds a NULL pointer check to avoid byte swapping those
block group descriptors in a bg descriptor block, but which are beyond
the end of the file system, to address this crash.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reported-by: Anatoly Pugachev <matorola@gmail.com>
10 months agolibcom_err: deal with the fact that the Hurd error messages are not zero-based
Theodore Ts'o [Wed, 8 Jan 2020 15:59:37 +0000 (10:59 -0500)]
libcom_err: deal with the fact that the Hurd error messages are not zero-based

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoMerge tag 'v1.45.5' into next
Theodore Ts'o [Tue, 7 Jan 2020 16:52:00 +0000 (11:52 -0500)]
Merge tag 'v1.45.5' into next

v1.45.5

10 months agoUpdate release notes, etc., for the 1.45.5 release
Theodore Ts'o [Mon, 6 Jan 2020 04:54:30 +0000 (23:54 -0500)]
Update release notes, etc., for the 1.45.5 release

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: always compile swapfs functions on all architectures
Theodore Ts'o [Mon, 6 Jan 2020 23:01:48 +0000 (18:01 -0500)]
libext2fs: always compile swapfs functions on all architectures

By only compiling the ext2fs_swap_* functions on big-endian systems,
it causes debian/libext2fs2.symbols to need to be different on
different little-endian vs big-endian architectures.  Including the
ext2fS_swap_* functions increases the size of the library by ~6k,
which is not a big deal.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoe2scrub, e2scrub_all: don't sleep unnecessarily in exitcode
Theodore Ts'o [Mon, 6 Jan 2020 21:01:23 +0000 (16:01 -0500)]
e2scrub, e2scrub_all: don't sleep unnecessarily in exitcode

The two second sleep is only needed in e2scrub, and when there is a
failure, so that systemd has a chance to gather the log output before
e2scrub exits.  It's not needed if the script is exiting successfully,
and it's never needed for e2scrub_all ever.

Addresses-Debian-Bug: #948193
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agodebian: add autopkgtest files
Theodore Ts'o [Mon, 6 Jan 2020 16:47:21 +0000 (11:47 -0500)]
debian: add autopkgtest files

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: don't scan /etc/mtab if file system not found in /proc/mounts
Theodore Ts'o [Mon, 6 Jan 2020 19:42:01 +0000 (14:42 -0500)]
libext2fs: don't scan /etc/mtab if file system not found in /proc/mounts

Previously we would scan /etc/mtab if the device is not found in
/proc/mounts.  This is because previously, /etc/mtab would have the
filename for a loopback mount, while /proc/mounts would only have
something like /dev/loop0.  Since on many systems /etc/mtab is now a
symlink to /proc/mounts, ismounted.c has a special function,
check_loop_mounted.

For this reason, it's not necessary to fall back to trying to scan
/etc/mtab if a device / filename is not found from scanning
/proc/mounts.  This also prevents failures if the file /etc/mtab does
not exist but /proc/mounts does exist when checking to see if a device
is mounted when it isn't.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoMerge branch 'maint' into next
Theodore Ts'o [Thu, 2 Jan 2020 15:59:13 +0000 (10:59 -0500)]
Merge branch 'maint' into next

10 months agoe2fsck: don't check for future superblock times if checkinterval == 0
Theodore Ts'o [Thu, 2 Jan 2020 02:50:27 +0000 (21:50 -0500)]
e2fsck: don't check for future superblock times if checkinterval == 0

We are no longer enabling periodic file system checks by default in
mke2fs.  The only reason why we force file system checks if the last
mount time or last write time in the superblock is if this might
bypass the periodic file systme checks.  So if the checkinterval is
zero, skip the last mount/write time checks since there's no reason to
force a check just because the system clock is incorrect.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agodebian: update debhelper compat level to 12
Theodore Ts'o [Thu, 2 Jan 2020 01:40:43 +0000 (20:40 -0500)]
debian: update debhelper compat level to 12

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoconfig: update config.{guess,sub}
Theodore Ts'o [Wed, 1 Jan 2020 20:16:41 +0000 (15:16 -0500)]
config: update config.{guess,sub}

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agopo: update zh_CN.po (from translationproject.org)
Boyuan Yang [Wed, 1 Jan 2020 19:33:29 +0000 (14:33 -0500)]
po: update zh_CN.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agopo: update ms.po (from translationproject.org)
Sharuzzaman Ahmat Raslan [Wed, 1 Jan 2020 19:33:29 +0000 (14:33 -0500)]
po: update ms.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoDrop use of -pedantic when doing gcc-wall
Theodore Ts'o [Wed, 1 Jan 2020 00:48:37 +0000 (19:48 -0500)]
Drop use of -pedantic when doing gcc-wall

With newer versions of gcc -pedantic is *super* pedantic, and
generates way too much noise.  So we drop it, and thus we don't need
util/gcc-wall-cleanup and util/static-analysis-cleanup.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: use ext2fs_file_llseek in inode_io.c
Theodore Ts'o [Wed, 1 Jan 2020 00:37:31 +0000 (19:37 -0500)]
libext2fs: use ext2fs_file_llseek in inode_io.c

Enable the use of files > 2GB when using the inode_io manager.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: chage ext2_off_t and ext2_off64_t to be signed types
Theodore Ts'o [Wed, 1 Jan 2020 00:00:49 +0000 (19:00 -0500)]
libext2fs: chage ext2_off_t and ext2_off64_t to be signed types

The ext2_off_t and ext2_off64_t types are used by ext2_file_lseek()
and ext2_file_llseek(), and they need to be signed so that it can be a
negative offset from the end of the file when EXT2_SEEK_END is used.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agolibext2fs: use __u64 instead of ext2_off64_t in qcow.c
Theodore Ts'o [Tue, 31 Dec 2019 22:16:25 +0000 (17:16 -0500)]
libext2fs: use __u64 instead of ext2_off64_t in qcow.c

The type ext2_off64_t should only be used in the context of the ext2fs
library's llseek() and ftruncate() analogs, ext2_file_llseek() and
ext2_inode_set_size().  That's because like the POSIX.1's off_t, it
needs to be a signed integer value so that SEEK_END will work
correctly.

qcow.c's use of ext2_off64_t is only internal, and not in any exposed
interfaces, so changes nothing; it just prepares us to be able to
change the type of ext2_off64_t.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agomke2fs: fix setting bad blocks in the block bitmap
Li Dongyang [Wed, 20 Nov 2019 04:35:24 +0000 (04:35 +0000)]
mke2fs: fix setting bad blocks in the block bitmap

We mark the bad blocks as used on fs->block_map before allocating
group tables.  Don't translate the block number to cluster number when
doing this, the fs->block_map is still a block-granularity allocation
map, it will be coverted later by ext2fs_convert_subcluster_bitmap().

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoe2image: remove redundant -fr from man page and usage message
Benno Schulenberg [Thu, 5 Dec 2019 17:57:35 +0000 (18:57 +0100)]
e2image: remove redundant -fr from man page and usage message

Also, add a missing dash and two missing brackets and two missing
spaces, and remove three excess spaces.

Signed-off-by: Benno Schulenberg <bensberg@telfort.nl>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoAOSP: e2fsdroid: Skip Base FS entries that no longer exist.
David Anderson [Sat, 7 Dec 2019 00:48:52 +0000 (16:48 -0800)]
AOSP: e2fsdroid: Skip Base FS entries that no longer exist.

Don't reserve blocks in the base map if the file does not exist in the
target image. This can happen if a file is removed or renamed in between
two builds. If the removed file is quite large, skipping it is important
since otherwise it will prevent blocks from being allocated for new files.

Bug: 145316683
Test: e2fsdroid with dynamic partitions
Change-Id: I63a9372c58adeaae3e1235fd92fed78a284ed391
From AOSP commit: 2b6646a3d2a7dd9972275cb829239ae033762da5

10 months agoAOSP: e2fsdroid: Allow re-use of deduplicated blocks.
David Anderson [Thu, 5 Dec 2019 22:00:28 +0000 (14:00 -0800)]
AOSP: e2fsdroid: Allow re-use of deduplicated blocks.

When using a Base FS map, track deduplicated blocks in a separate
bitmap. The first inode to request a block from this set will succeed
in getting the block. Blocks in the dedup set are not available for
libext2fs to allocate; this ensures that previously deduplicated blocks
are re-used for their original purpose.

Note that deduplication takes priority over block allocation, so that
once a block is removed from the dedup set, that does not actually
prevent it from being re-used. Similarly, a file that was not previously
sharing a block may have its blocks shared in the new image.

Bug: 145316683
Test: e2fsdroid with dynamic partitions
Change-Id: I73856faa5d294a7b5fb985ccd1a6974a989481ea
From AOSP commit: 4e55425ff5d7d7cea27dcf79125766762e2b3529

10 months agoAOSP: e2fsdroid: Do not allocate invalid blocks from BaseFS.
David Anderson [Thu, 5 Dec 2019 21:27:44 +0000 (13:27 -0800)]
AOSP: e2fsdroid: Do not allocate invalid blocks from BaseFS.

If certain metadata properties change in between builds, such as the
inode table size, then block mappings may not be valid from one build to
the next. For example, build A could allocate block N for a file. If
build B has a larger inode table, block N may no longer be a data block.

In this case, we need to remove the block from the BaseFS range list so
we do not give it back to libext2fs for writing data. We do this with a
new "owned_block_map" bitmap. If a block from the base FS is not in use
by the initial image, and is not used by any other file, it is
considered owned, and can be claimed by that file.

Note that while this produces correct images, it also prevents
deduplicated blocks from being re-used. This will be addressed in a
follow-up patch.

Bug: 145316683
Test: e2fsdroid block mapping with dynamic partitions
Change-Id: I3145e45156f7879bdf956384723fab4bd69acb93
Signed-off-by: David Anderson <dvander@google.com>
From AOSP commit: b3d1ccdb673772588b7bb14c4581980ef549a0b8

10 months agoAOSP: e2fsdroid: Refactor basefs allocation.
David Anderson [Thu, 5 Dec 2019 20:23:12 +0000 (12:23 -0800)]
AOSP: e2fsdroid: Refactor basefs allocation.

This refactors base_fs_alloc_load() to address two issues. First, the
failure cases have been made simpler by factoring out a common helper
for freeing a base_fs_allocator. Second, we no longer return
EXIT_FAILURE, since this is not an errcode_t.

Bug: 145316683
Change-Id: I8b28c684ecf3423b605b550177d7ead4eb2821a4
From AOSP commit: b461ade872a809ab913d7506e9b8ed597d57dc87

10 months agoAOSP: e2fsdroid: Refactor block_range.
David Anderson [Thu, 5 Dec 2019 01:38:11 +0000 (17:38 -0800)]
AOSP: e2fsdroid: Refactor block_range.

block_range is a singly-linked list, but the head/tail links are
manually managed all over. Instead, introduce a block_range_list
structure and refactor list helpers to operate on this instead. This
ensures head/tail are maintained properly (in some cases, like
delete_block_range, they were not).

Bug: 145316683
Test: manual test
Change-Id: Ieec6324549e2c3a71129871f703f4f0a37aeb1fa
From AOSP commit: 4220594792297619d2e70a29476667d1698dbd63

10 months agoAOSP: Add vendor_available: true
Justin Yun [Fri, 6 Dec 2019 06:14:36 +0000 (15:14 +0900)]
AOSP: Add vendor_available: true

Make libext2-headers and libext2_uuid available to vendor and
product. They are required by f2fs-tools which are installed in
product partition.

Bug: 120954888
Bug: 145580164
Test: set PRODUCT_PRODUCT_VNDK_VERSION := current and build
Change-Id: Ibf5678448f9efbcb72c0544a3a50280fbfee2f38
From AOSP commit: 2c79a5f79197c090fc9c9534546f4ab4f7690809

10 months agoe2fsck: fix use after free in calculate_tree()
Wang Shilong [Tue, 31 Dec 2019 00:52:39 +0000 (19:52 -0500)]
e2fsck: fix use after free in calculate_tree()

The problem is alloc_blocks() will call get_next_block() which might
reallocate outdir->buf, and memory address could be changed after
this.  To fix this, pointers that point into outdir->buf, such as
int_limit and root need to be recaulated based on the new starting
address of outdir->buf.

[ Changed to correctly recalculate int_limit, and to optimize how we
  reallocate outdir->buf.  -TYT ]

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoe2fsck: fix to return ENOMEM in alloc_size_dir()
Wang Shilong [Tue, 26 Nov 2019 09:03:58 +0000 (18:03 +0900)]
e2fsck: fix to return ENOMEM in alloc_size_dir()

Two memory allocation return check is missed.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 months agoext2fs: add ext2fs_get_stat_i_blocks() function
Theodore Ts'o [Mon, 30 Dec 2019 15:12:58 +0000 (10:12 -0500)]
ext2fs: add ext2fs_get_stat_i_blocks() function

The function ext2fs_inode_i_blocks() is a bit confusing whether it is
returning the inode's i_blocks value, or whether it is returning the
value ala the stat(2) system call, which returns i_blocks in units of
512 byte sectors.  This caused ext2fs_inode_i_blocks() to be
incorrectly used in fuse2fs and the function quota_compute_usage().

To address this, we add a new function, ext2fs_get_stat_i_blocks()
which is clearly labelled what it is returning, and use it in fuse2fs
and quota_compute_usage().  It's also a bit more convenient to use it
in e2fsck, so use it there too.

Reported-by: Wang Shilong <wangshilong1991@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agolibext2fs: fix crash in ext2fs_open2() on Big Endian systems
Theodore Ts'o [Fri, 27 Dec 2019 04:19:54 +0000 (23:19 -0500)]
libext2fs: fix crash in ext2fs_open2() on Big Endian systems

Commit e6069a05: ("Teach ext2fs_open2() to honor the
EXT2_FLAG_SUPER_ONLY flag") changed how the function
ext2fs_group_desc() handled a request for a gdp pointer for a group
larger than the number of groups in the file system; it now returns
NULL, instead of returning a pointer beyond the end of the array.

Previously, the ext2fs_open2() function would swap all of the block
group descriptors in a block, even if they are beyond the end of the
file system.  This was OK, since we were not overrunning the allocated
memory, since it was rounded to a block boundary.  But now that
ext2fs_group_desc() would return NULL for those gdp, it would cause
ext2fs_open2(), when it was byte swapping the block group descriptors
on Big Endian systems, to dereference a null pointer and crash.

This commit adds a NULL pointer check to avoid byte swapping those
block group descriptors in a bg descriptor block, but which are beyond
the end of the file system, to address this crash.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reported-by: Anatoly Pugachev <matorola@gmail.com>
11 months agofuse2fs: add support for 32-bit uids and gids
Theodore Ts'o [Fri, 20 Dec 2019 20:16:09 +0000 (15:16 -0500)]
fuse2fs: add support for 32-bit uids and gids

Previously, uids were truncated at 16 bits because we weren't properly
handling i_uid_high and i_gid_high.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agomke2fs: fix "mke2fs -d /path/to/files" to support 32-bit uids and gids
Theodore Ts'o [Fri, 20 Dec 2019 19:58:41 +0000 (14:58 -0500)]
mke2fs: fix "mke2fs -d /path/to/files" to support 32-bit uids and gids

https://github.com/tytso/e2fsprogs/issues/29

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agoe2fsck: don't try to rehash a deleted directory
Theodore Ts'o [Fri, 20 Dec 2019 00:45:06 +0000 (19:45 -0500)]
e2fsck: don't try to rehash a deleted directory

If directory has been deleted in pass1[bcd] processing, then we
shouldn't try to rehash the directory in pass 3a when we try to
rehash/reoptimize directories.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agoe2fsck: abort if there is a corrupted directory block when rehashing
Theodore Ts'o [Fri, 20 Dec 2019 00:37:34 +0000 (19:37 -0500)]
e2fsck: abort if there is a corrupted directory block when rehashing

In e2fsck pass 3a, when we are rehashing directories, at least in
theory, all of the directories should have had corruptions with
respect to directory entry structure fixed.  However, it's possible
(for example, if the user declined a fix) that we can reach this stage
of processing with a corrupted directory entries.

So check for that case and don't try to process a corrupted directory
block so we don't run into trouble in mutate_name() if there is a
zero-length file name.

Addresses: TALOS-2019-0973
Addresses: CVE-2019-5188
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agoe2fsck: optimize away repeated calls to gettext()
Theodore Ts'o [Sat, 14 Dec 2019 04:30:53 +0000 (23:30 -0500)]
e2fsck: optimize away repeated calls to gettext()

Optimize _("getting next inode from scan") so it is not called for
each initialized inode in the file system, and make a similar
optimization in pass 2 for each directory block.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
11 months agoe2scrub: fix the 30 second timeout when trying to remove a snapshot
Theodore Ts'o [Sat, 14 Dec 2019 04:23:06 +0000 (23:23 -0500)]
e2scrub: fix the 30 second timeout when trying to remove a snapshot

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
12 months agoMerge branch 'maint' into next
Theodore Ts'o [Sat, 23 Nov 2019 04:30:37 +0000 (23:30 -0500)]
Merge branch 'maint' into next

12 months agoAdd a program to test images provided by UBSAN fuzzing reports
Theodore Ts'o [Sat, 23 Nov 2019 04:27:47 +0000 (23:27 -0500)]
Add a program to test images provided by UBSAN fuzzing reports

This program calls a few ext2fs library functions used by the current
generation of libext2fs fuzzers, and is helpful in reproducing UBSAN
failures reported externally.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
12 months agolibext2fs: define PATH_MAX if not provided by the system headers
Theodore Ts'o [Sat, 23 Nov 2019 04:26:39 +0000 (23:26 -0500)]
libext2fs: define PATH_MAX if not provided by the system headers

This is needed to compile on Illumos and its derivatives.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>