Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Allow servers to use a buffer pool that is also registered through io-uring. When the server registers a buffer pool with io-uring, the pages backing the pool are pinned upfront. This eliminates the overhead of pinning/unpinning user pages and translating virtual addresses per i/o request. This also allows servers to use the same registered memory for subsequent backing store I/O (eg read_fixed/write_fixed), keeping data in the same pinned pages without additional pinning or mapping overhead required. To use this, the server needs to set the FUSE_URING_REGISTERED_BUFPOOL flag when adding a bufpool through the FUSE_IO_URING_CMD_ADD_BUFPOOL cmd. For every sqe submitted (including the one for adding the bufpool), it should set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED, and pass in the index where the registered bufpool resides to sqe->buf_index. Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a 2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where direct-I/O throughput is against a RAM-backed (tmpfs) source (backing I/O is not the bottleneck): baseline registered buffers direct read ~5.1 GB/s ~5.4 GB/s (+~5%) direct write ~3.4 GB/s ~4.8 GB/s (+~45%) Registered buffers bring up the write path speed up closer to speed of reads. There isn't much improvement for reads because it is already fast enough where it's at the copy-bound ceiling (surpassing that requires doing zero-copy). On a device-bound NVMe though, the differences are within noise, as backing I/O dominates per-request latency. Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Reviewed-by: Bernd Schubert <bernd@bsbernd.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Right now, ents and buffers are tightly coupled in fuse io-uring where
each entry has its own dedicated payload buffer, requiring N buffers for
N entries where each buffer must be large enough to accomodate the
maximum payload size. This is suboptimal as most request types (lookup,
open, release, getattr, etc) require vastly less bytes than the maximum
payload size and some requests (unlink, rmdir, fsync, flush, etc) do not
require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers,
allow the server to pass in a buffer pool (a contiguous chunk of memory)
that the kernel will use as it wishes for servicing ents/requests.
Entries only reserve a "buffer" from the pool while actively processing
a request that requires a payload buffer. This decoupling and letting
the kernel delegate memory from the pool for requests allows the kernel to
optimize memory usage and reduces the memory usage requirements needed
to use fuse-over-io-uring.
A pool is registered per queue with the new
FUSE_IO_URING_CMD_ADD_BUFPOOL command. The server passes the pool's base
address and length in fuse_uring_cmd_req.bufpool.{uaddr,len}.
Internally, the kernel splits the region into buffers of
ring->max_payload_sz bytes each (nr_bufs = pool len / max_payload_sz). A
queue commits to a payload mode on first use: registering an entry that
carries its own payload selects the legacy per-entry mode, while
ADD_BUFPOOL selects pool mode. The two are mutually exclusive, so
ADD_BUFPOOL must be issued before any payload-carrying entries are
registered on that queue. The queue must have been created before the
bufpool is added, through the FUSE_IO_URING_CMD_ADD_QUEUE command.
The kernel tracks free buffers with a bitmap (a set bit marks a free
buffer). On dispatch, a request that needs a payload claims a free
buffer (find_first_bit + clear). A request that needs none claims
nothing. The buffer's byte offset within the pool is reported to the
server in the new fuse_uring_ent_in_out.offset field so that the server
can locate the payload. On completion the buffer is returned to the pool
or reused directly if the next request on that entry also has a payload.
The FUSE_HAS_IO_URING_BUFPOOL flag advertises kernel support to the
server for bufpools.
Buffer pool request flow
~~~~~~~~~~~~~~~~~~~~~~~~
| Kernel | FUSE daemon
| |
| [request arrives] |
| [claim a free pool buffer] |
| >fuse_uring_select_buffer() |
| [copy headers to ring] |
| [copy payload to buffer] |
| [report buffer offset in ent_in_out] |
| >io_uring_cmd_done() |
| | [read headers]
| | [read/write payload at offset]
| | [process request]
| | >io_uring_submit()
| | COMMIT_AND_FETCH
| >fuse_uring_commit_fetch() |
| [copy reply from ring] |
| [return buffer to the pool] |
| >fuse_uring_recycle_buffer() |
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Currently, the connection's fuse_ring is created lazily on the first FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one thread per queue (one per CPU) and those threads issue their first REGISTER command concurrently. They then race to create the single per-connection fuse_ring, which required open-coded handling in fuse_uring_create() to detect and protect against concurrent creations. Decouple fuse_ring creation from ent registration and move it to FUSE_INIT reply processing after a server has negotiated and set FUSE_OVER_IO_URING. The ring is published before the connection is marked initialized. fuse_uring_register() no longer creates the ring and it instead uses the ring set up at init time. Reviewed-by: Bernd Schubert <bernd@bsbernd.com> Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Use regular C comment syntax for a non-kernel-doc comment to avoid a kernel-doc warning: Warning: fs/fuse/dev_uring_i.h:104 This comment starts with '/**', but isn't a kernel-doc comment. * Describes if uring is for communication and holds alls the data needed Binary build output is the same before and after this change. Signed-off-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Store pointer to struct fuse_chan instead of struct fuse_conn in fuse_ring. Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Start getting rid of fs layer stuff from transport layer files. Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Add helpers (get and set functions mainly) that cleanly separate the layers. Remove #include "fuse_dev_i.h" from: - inode.c - file.c - control.c Remove #include "dev_uring_i.h" from inode.c. [Li Wang: drop redundant initializer in process_init_limits()] Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Move: - timeout Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Move: - io_uring - ring Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>