linux.git/fs/verity, branch v6.3

fsverity: don't drop pagecache at end of FS_IOC_ENABLE_VERITY

2023-03-16T05:50:41+00:00

The full pagecache drop at the end of FS_IOC_ENABLE_VERITY is causing
performance problems and is hindering adoption of fsverity.  It was
intended to solve a race condition where unverified pages might be left
in the pagecache.  But actually it doesn't solve it fully.

Since the incomplete solution for this race condition has too much
performance impact for it to be worth it, let's remove it for now.

Fixes: 3fda4c617e84 ("fs-verity: implement FS_IOC_ENABLE_VERITY ioctl")
Cc: stable@vger.kernel.org
Reviewed-by: Victor Hsieh 
Link: https://lore.kernel.org/r/20230314235332.50270-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers

fsverity: Remove WQ_UNBOUND from fsverity read workqueue

2023-03-14T23:16:36+00:00

WQ_UNBOUND causes significant scheduler latency on ARM64/Android.  This
is problematic for latency sensitive workloads, like I/O
post-processing.

Removing WQ_UNBOUND gives a 96% reduction in fsverity workqueue related
scheduler latency and improves app cold startup times by ~30ms.
WQ_UNBOUND was also removed from the dm-verity workqueue for the same
reason [1].

This code was tested by running Android app startup benchmarks and
measuring how long the fsverity workqueue spent in the runnable state.

Before
Total workqueue scheduler latency: 553800us
After
Total workqueue scheduler latency: 18962us

[1]: https://lore.kernel.org/all/20230202012348.885402-1-nhuck@google.com/

Signed-off-by: Nathan Huckleberry 
Fixes: 8a1d0f9cacc9 ("fs-verity: add data verification hooks for ->readpages()")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/20230310193325.620493-1-nhuck@google.com
Signed-off-by: Eric Biggers

fsverity: support verifying data from large folios

2023-01-27T22:46:31+00:00

Try to make fs/verity/verify.c aware of large folios.  This includes
making fsverity_verify_bio() support the case where the bio contains
large folios, and adding a function fsverity_verify_folio() which is the
equivalent of fsverity_verify_page().

There's no way to actually test this with large folios yet, but I've
tested that this doesn't cause any regressions.

Signed-off-by: Eric Biggers 
Link: https://lore.kernel.org/r/20230127221529.299560-1-ebiggers@kernel.org

fsverity: support enabling with tree block size < PAGE_SIZE

2023-01-10T03:06:06+00:00

Make FS_IOC_ENABLE_VERITY support values of
fsverity_enable_arg::block_size other than PAGE_SIZE.

To make this possible, rework build_merkle_tree(), which was reading
data and hash pages from the file and assuming that they were the same
thing as "blocks".

For reading the data blocks, just replace the direct pagecache access
with __kernel_read(), to naturally read one block at a time.

(A disadvantage of the above is that we lose the two optimizations of
hashing the pagecache pages in-place and forcing the maximum readahead.
That shouldn't be very important, though.)

The hash block reads are a bit more difficult to handle, as the only way
to do them is through fsverity_operations::read_merkle_tree_page().

Instead, let's switch to the single-pass tree construction algorithm
that fsverity-utils uses.  This eliminates the need to read back any
hash blocks while the tree is being built, at the small cost of an extra
block-sized memory buffer per Merkle tree level.  This is probably what
I should have done originally.

Taken together, the above two changes result in page-size independent
code that is also a bit simpler than what we had before.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-8-ebiggers@kernel.org

fsverity: support verification with tree block size < PAGE_SIZE

2023-01-10T03:06:03+00:00

Add support for verifying data from verity files whose Merkle tree block
size is less than the page size.  The main use case for this is to allow
a single Merkle tree block size to be used across all systems, so that
only one set of fsverity file digests and signatures is needed.

To do this, eliminate various assumptions that the Merkle tree block
size and the page size are the same:

- Make fsverity_verify_page() a wrapper around a new function
  fsverity_verify_blocks() which verifies one or more blocks in a page.

- When a Merkle tree block is needed, get the corresponding page and
  only verify and use the needed portion.  (The Merkle tree continues to
  be read and cached in page-sized chunks; that doesn't need to change.)

- When the Merkle tree block size and page size differ, use a bitmap
  fsverity_info::hash_block_verified to keep track of which Merkle tree
  blocks have been verified, as PageChecked cannot be used directly.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-7-ebiggers@kernel.org

fsverity: replace fsverity_hash_page() with fsverity_hash_block()

2023-01-10T03:06:01+00:00

In preparation for allowing the Merkle tree block size to differ from
PAGE_SIZE, replace fsverity_hash_page() with fsverity_hash_block().  The
new function is similar to the old one, but it operates on the block at
the given offset in the page instead of on the full page.

(For now, all callers still pass a full page.)

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-6-ebiggers@kernel.org

fsverity: use EFBIG for file too large to enable verity

2023-01-10T03:05:59+00:00

Currently, there is an implementation limit where files can't have more
than 8 Merkle tree levels.  With SHA-256 and 4K blocks, this limit is
never reached, since a file would need to be larger than 2**64 bytes to
need 9 levels.  However, with SHA-512, 9 levels are needed for files
larger than about 1.15 EB, which is possible on btrfs.  Therefore, this
limit technically became reachable when btrfs added fsverity support.

Meanwhile, support for merkle_tree_block_size < PAGE_SIZE will introduce
another implementation limit on file size, resulting from the use of an
in-memory bitmap to track which Merkle tree blocks have been verified.

In any case, currently FS_IOC_ENABLE_VERITY fails with EINVAL when the
file is too large.  This is undocumented, and also ambiguous since
EINVAL can mean other things too.  Let's change the error code to EFBIG,
which is much clearer, and document it.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-5-ebiggers@kernel.org

fsverity: store log2(digest_size) precomputed

2023-01-10T03:05:56+00:00

Add log_digestsize to struct merkle_tree_params so that it can be used
in verify.c.  Also save memory by using u8 for all the log_* fields.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-4-ebiggers@kernel.org

fsverity: simplify Merkle tree readahead size calculation

2023-01-10T03:05:54+00:00

First, calculate max_ra_pages more efficiently by using the bio size.

Second, calculate the number of readahead pages from the hash page
index, instead of calculating it ahead of time using the data page
index.  This ends up being a bit simpler, especially since level 0 is
last in the tree, so we can just limit the readahead to the tree size.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-3-ebiggers@kernel.org

fsverity: use unsigned long for level_start

2023-01-10T03:05:47+00:00

fs/verity/ isn't consistent with whether Merkle tree block indices are
'unsigned long' or 'u64'.  There's no real point to using u64 for them,
though, since (a) a Merkle tree with over ULONG_MAX blocks would only be
needed for a file larger than MAX_LFS_FILESIZE, and (b) for reads, the
status of all Merkle tree blocks has to be tracked in memory.

Therefore, let's make things a bit more efficient on 32-bit systems by
using 'unsigned long[]' for merkle_tree_params::level_start, instead of
'u64[]'.  Also, to be extra safe, explicitly check that there aren't
more than ULONG_MAX Merkle tree blocks.

Signed-off-by: Eric Biggers 
Reviewed-by: Andrey Albershteyn 
Tested-by: Ojaswin Mujoo 
Link: https://lore.kernel.org/r/20221223203638.41293-2-ebiggers@kernel.org