fuse_get_req() parks background allocations on fch->blocked_waitq via wait_event_state_exclusive(), so each wakeup releases exactly one waiter. fuse_chan_max_background_set() clears fch->blocked when the new limit exceeds num_background, but the accompanying wake_up() releases a single waiter regardless of how many slots just became available. Raising max_background from 10 to 100 therefore admits one request instead of ninety. The remaining waiters are not permanently stranded — the "else if (!fch->blocked)" branch in fuse_request_end() wakes one more per completion — but that only helps while requests keep completing. Consider a fixed pool of threads doing readahead or async direct I/O with the quota exhausted: every thread is either in flight or parked, and each completion wakes one waiter while freeing one slot, a net change of zero. num_background oscillates around the old limit and the added quota is never taken up. Waking one waiter per freed slot also preserves submission order: once fch->blocked is clear, new callers of fuse_get_req() skip the waitqueue entirely, overtaking waiters that parked before the limit was raised. Use wake_up_nr() with the number of slots that just became available. Since the wakeup is guarded by !fch->blocked, num_background is strictly below max_background, so the count is at least 1 and never degenerates into wake_up_all(). Signed-off-by: Baokun Li <libaokun@linux.alibaba.com> Reviewed-By: Horst Birthelmer <hbirthelmer@ddn.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
The abort_on_kill path in request_wait_answer() calls fuse_abort_conn()
and returns without waiting for FR_FINISHED. If fuse_dev_do_write() is
concurrently processing the same request (FR_LOCKED set), the caller
frees req->args while it is still being accessed, causing a
use-after-free.
Fix this by jumping to the existing wait_event(FR_FINISHED) instead of
returning early. The wait will not hang because fuse_abort_conn()
ensures all requests are ended.
Reported-by: syzbot+d6540a3fa1626e11360d@syzkaller.appspotmail.com
Fixes: 204aa22a686b ("fuse: abort on fatal signal during sync init")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Rochan Avlur <rochan.avlur@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Currently, the connection's fuse_ring is created lazily on the first FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one thread per queue (one per CPU) and those threads issue their first REGISTER command concurrently. They then race to create the single per-connection fuse_ring, which required open-coded handling in fuse_uring_create() to detect and protect against concurrent creations. Decouple fuse_ring creation from ent registration and move it to FUSE_INIT reply processing after a server has negotiated and set FUSE_OVER_IO_URING. The ring is published before the connection is marked initialized. fuse_uring_register() no longer creates the ring and it instead uses the ring set up at init time. Reviewed-by: Bernd Schubert <bernd@bsbernd.com> Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_chan_set_initialized() sets values for the connection state and then sets fch->initialized to true, but lockless readers read fch->initialized and if true, go to read the connection state values, without using any barriers. There are a few instances where this happens (fuse_uring_cmd() before dispatching register / commit-and-fetch cmds, fuse_dev_do_wriite() for handling notify retrieves, etc). To make this as simple as possible, use release/acquire semantics for writing/reading fch->initialized. Add the missing read barriers. This is not marked for stable as these are not realistically reachable on a well-behaved server, and buggy/malicious servers who trigger this path fail benignly rather than crash or deadlock the kernel. Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_block_alloc() reads fch->initialized and then fch->io_uring.
fch->io_uring is set before fch->initialized, ordered by the smp_wmb()
in fuse_chan_set_intialized(), but fuse_block_alloc() has no matching
read barrier between the two loads.
This may lead a CPU to observe fch->initialized=1 but fch->io_uring=0,
and skip the check that blocks request allocation until the io-uring
queues are ready. This can reintroduce the lock-order inversion deadlock
that commit 3393ff964e0f prevents.
Add an smp_rmb() barrier to pair with the smp_wmb() in
fuse_chan_set_initialized() to prevent this.
Fixes: 3393ff964e0f ("fuse: block request allocation until io-uring init is complete")
Cc: stable@vger.kernel.org
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
After commit f8fce75fedf7 ("fuse: clear intr_entry in fuse_resend and
fuse_remove_pending_req") the WARN_ON(!list_empty(&req->intr_entry)) in
fuse_request_free() still triggers due to the following race:
In request_wait_answer()
if (test_bit(FR_SENT, &req->flags)) -> returns true
In fuse_chan_resend()
clear_bit(FR_SENT, &req->flags)
In request_wait_answer()
queue_interrupt(req)
Fix by:
- move clearing FR_SENT inside fpq->lock
- move setting FR_PENDING inside fiq->lock
- recheck FR_SENT after acquiring fiq->lock in fuse_dev_queue_interrupt()
Reported-by: zdi-disclosures@trendmicro.com
Fixes: f8fce75fedf7 ("fuse: clear intr_entry in fuse_resend and fuse_remove_pending_req")
Cc: stable@vger.kernel.org # 6.9
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Clean up interrupt reading logic. Remove passing the pointer to the fuse request as an arg and make the header initializations more readable. Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Remove stray newline that shouldn't be there. Signed-off-by: Joanne Koong <joannelkoong@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>