commit c25aee9c630fb86f98d79eccb75765067079b972 upstream.
ceph_dirty_folio() takes a wrbuffer claim on each newly dirtied folio: it
bumps i_wrbuffer_ref (taking an ihold() on the 0->1 transition) and
attaches the snap_context to folio->private. That claim is released only
by ceph_put_wrbuffer_cap_refs(), which for a submitted write runs from
writepages_finish().
In ceph_submit_write(), if ceph_inc_osd_stopping_blocker() fails -- which
happens during umount -- the request is aborted before submission: the
already-collected folios are only redirtied and unlocked, so
writepages_finish() never runs and the claim is leaked.
redirty_page_for_writepage() -> folio_redirty_for_writepage() ->
filemap_dirty_folio() sets PG_dirty directly and does not go through
->dirty_folio, so ceph_dirty_folio() is not re-entered to rebalance it.
Because every subsequent writeback also fails the osd_stopping_blocker,
i_wrbuffer_ref never returns to 0, the ihold() is never dropped, and the
inode cannot be evicted:
VFS: Busy inodes after unmount of ceph
kernel BUG at fs/super.c:650!
Release the orphaned claim in the abort path before redirtying, via
ceph_undo_wrbuffer_claim(): detach the snap_context, drop the wrbuffer
reference (letting i_wrbuffer_ref reach 0 and iput() the inode), and drop
the snap_context reference -- i.e. do what writepages_finish() would have
done for these never-submitted folios.
Only the locked_pages entries are undone; folios still in the fbatch were
never dirty-cleared by this call (folio_clear_dirty_for_io() is the
ownership-transfer point, and a successful move NULLs the fbatch slot), so
they hold no claim this call owns.
Cc: stable@vger.kernel.org
Fixes: fd7449d937e7 ("ceph: fix generic/421 test failure")
Signed-off-by: Matthew Brown <matthew@bargrove.com>
Reviewed-by: Xiubo Li <xiubo.li@clyso.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit e7d7aa7b730178278109c41fa1b17b06873065d5 upstream.
ceph_cap_reclaim_work() re-queues itself for as long as
ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease
walk exhausts its `nr_to_scan` budget. This creates a busy loop that
consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (`count==0`) and every scanned lease
still valid, each pass runs the full scan budget down to zero and
returns `-EAGAIN`, only to be queued again immediately.
The dir-lease walk made this worse. When `expire_dir_lease` is
`false` (i.e. we have no intention of reclaiming dir leases),
__dir_lease_check() returned `TOUCH` for every valid lease. `TOUCH`
moves the dentry to the tail of the list and resets `di->time` via
__dentry_dir_lease_touch(), so a walk over N valid leases pointlessly
rewrote the list, refreshed the timestamps (preventing them from ever
aging out) and always drained `nr_to_scan`, guaranteeing the `-EAGAIN`
requeue.
Fix this in three steps:
- Return `KEEP` instead of `TOUCH` when `expire_dir_lease` is
`false`. If we are not going to reclaim the lease, leave it in
place instead of churning the list and resetting its timestamp; the
walk then terminates naturally (or via `STOP` at the first fresh
lease).
- Only return `-EAGAIN` from the first (dentry-lease) walk when something
was actually freed. A full batch that frees nothing means retrying
the same list immediately is futile; fall through to the dir-lease
walk instead.
- After both walks, bail out with success (0) when nothing was freed
and there is no cap pressure (`count==0`). There is no reason to
keep retrying when we are not over the cap limit and made no
progress.
Under real cap pressure (`count>0`) the reclaim path is unchanged and
still retries via `-EAGAIN`.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on
our web servers. This is very visible in `/proc/lock_stat` (5 minute
capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------
&mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
-----------------------
&mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------
&mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
-----------------------
&mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
__dentry_leases_walk() is almost gone. The total wait time is reduced
by a factor of 483. That will give some latency gains to
ceph_readdir().
Cc: stable@vger.kernel.org
Fixes: 37c4efc1ddf9 ("ceph: periodically trim stale dentries")
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Reviewed-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 68d541754d6cd3bb98d1fd8314f57e5eb533557d upstream.
__build_xattrs() decodes the MDS-supplied xattr blob one attribute at a
time. For each attribute it reads a 32-bit name length, advances past the
name bytes, reads a 32-bit value length, records the value pointer, and
advances past the value bytes. The two length fields are read with
ceph_decode_32_safe(), but the value bytes themselves are advanced over
with a bare "p += len" and no ceph_decode_need() check that "len" bytes
remain in the blob.
For every attribute except the last, the next iteration's
ceph_decode_32_safe() on the following name length implicitly verifies
that the previous value did not run past the blob end. The final
attribute has no successor, so its decoded value length is never checked
against the blob bounds. A malicious or compromised metadata server can
set the last attribute's value length larger than the bytes actually
present in the blob.
The blob is a dedicated kvmalloc() allocation sized to the wire length
(ceph_buffer_new() in ceph_fill_inode()). __set_xattr() records the
oversized length in xattr->val_len verbatim, and a later getxattr(2) runs
memcpy(value, xattr->val, xattr->val_len) into a user-supplied buffer,
copying bytes past the end of the allocation back to user space.
Impact: a malicious metadata server discloses adjacent kernel heap bytes
to a local user via getxattr(2) on a CephFS file. Add the missing
ceph_decode_need() so an out-of-bounds value length on the final
attribute fails the decode and returns -EIO instead of being stored.
Cc: stable@vger.kernel.org
Fixes: 355da1eb7a1f ("ceph: inode operations")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit a3eb169ee297aa99670ba927c659990bd1e453f3 upstream.
ceph_mdsmap_decode() in fs/ceph/mdsmap.c reads num_export_targets from
each per-mds info record and advances the decode cursor by
num_export_targets * sizeof(u32) without first checking that many bytes
remain. The only upper-bound check that catches a runaway cursor
(*p > info_end) is gated on info_v >= 4, because info_end is left NULL
for info_v 2 and 3. When the monitor sends an MDS map whose per-mds
info version is 2 or 3 with an oversized num_export_targets, the cursor
moves past the message front buffer and the later export-targets loop
calls the unchecked ceph_decode_32() on out-of-bounds memory.
A kernel client processes CEPH_MSG_MDS_MAP from its monitor session
(net/ceph/mon_client.c dispatches it; fs/ceph/super.c routes it to
ceph_mdsc_handle_mdsmap(), which sets end to the front buffer bound and
calls ceph_mdsmap_decode()). A malicious or compromised monitor, or an
on-path attacker on an unsigned/unencrypted messenger session, can
therefore drive an out-of-bounds read in the client kernel; on x86_64
with KASAN it is reported as a slab-out-of-bounds read in
ceph_mdsmap_decode(). The decoded values land in the internal
info->export_targets[] array, so the consequence is a kernel
out-of-bounds read, not an information leak to the attacker.
Impact: a malicious or compromised Ceph monitor sending an MDS map with
a per-mds info version of 2 or 3 and an oversized num_export_targets
field triggers an out-of-bounds read in the CephFS client kernel.
Add a ceph_decode_need() for the export-targets array before advancing
the cursor, so the bound is enforced for every info_v >= 2, not only
info_v >= 4. This mirrors the count-then-need idiom already used for
m_data_pg_pools later in the same function.
Compute the export-targets byte count with size_mul() and reuse that
checked length when advancing the cursor, so the attacker-controlled
num_export_targets multiplication fails closed on overflow rather than
relying on the later kcalloc() guard.
Cc: stable@vger.kernel.org
Fixes: d463a43d69f4 ("ceph: CEPH_FEATURE_MDSENC support")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 77933e22adfe813be2bd10be08d6e950103c3967 upstream.
handle_session() decodes the MDSCapAuth records carried by a
CEPH_SESSION_OPEN message (msg_version >= 6). For each record the
match.path and match.fs_name byte strings are read by first decoding a
32-bit length and then copying that many bytes with the bare
ceph_decode_copy(). Unlike the surrounding fields, which all use the
_safe decode variants, these two copies are not preceded by a
ceph_decode_need() bounds check, and the enclosing MDSCapAuth and
MDSCapMatch struct_len fields are skipped rather than enforced as an
upper bound. A length larger than the bytes remaining in the message
front makes ceph_decode_copy() read past the end of the front buffer.
The message front is a dedicated allocation (ceph_msg_new2() ->
kvmalloc), so the over-read runs off that object. A malicious or
compromised MDS can trigger this with the first post-connect message on
mount, with no client-side user interaction; under KASAN it is reported
as a slab-out-of-bounds read in handle_session().
Impact: a malicious MDS can force the kernel client to read up to 4 GiB
past the message front allocation during session setup, crashing the
client (out-of-bounds read).
Switch both copies to ceph_decode_copy_safe(), which performs the
ceph_decode_need() bounds check before the copy and branches to the
existing bad label, matching the rest of the decoder and the error path
that frees the partially decoded cap_auths array.
Cc: stable@vger.kernel.org
Fixes: 1d17de9534cb ("ceph: save cap_auths in MDS client when session is opened")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Reviewed-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit eff8013c5a8916613c742ae5a2cc341cb605c0ae upstream.
ceph_get_name() copies the MDS-supplied name into the caller's
NAME_MAX-sized buffer with memcpy(name, rinfo->dname, rinfo->dname_len)
and then writes name[rinfo->dname_len] = 0, without checking dname_len
against NAME_MAX. A malicious or buggy MDS that returns a LOOKUPNAME reply
with dname_len > NAME_MAX overflows the buffer. __get_snap_name() copies
rde->name / rde->name_len the same unchecked way.
Impact: a malicious or compromised Ceph MDS overflows the NAME_MAX name
buffer in a client's NFS-export get_name path, a slab out-of-bounds write
reported by KASAN. Reachable when a CephFS mount is re-exported over NFS.
Add ceph_export_copy_name(), which rejects lengths above NAME_MAX with
-ENAMETOOLONG before the copy, and use it in both ceph_get_name() and
__get_snap_name().
Cc: stable@vger.kernel.org
Fixes: 19913b4eac4a ("ceph: add get_name() NFS export callback")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit aedc9053d909508a5f56c3f49f885fc030df4730 upstream.
MDSMap export_targets entries are monitor controlled. check_new_map()
uses each entry as a bit number in a fixed stack bitmap, so a rank
outside the protocol namespace can make set_bit() write past the end of
the array.
Reject ranks outside CEPH_MAX_MDS while decoding the map. Do not
validate against possible_max_rank here because maps may legitimately
reference ranks beyond a temporarily reduced max_mds.
Cc: stable@vger.kernel.org
Fixes: d517b3983dd3 ("ceph: reconnect to the export targets on new mdsmaps")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 7af4c4f01305b0935adf6d4301b1ec407025485d upstream.
list_for_each_entry() iterates ci->i_cap_flush_list but drops
i_ceph_lock to send cap messages. During the unlock window,
handle_cap_flush_ack() can acquire i_ceph_lock, detach cf entries
with tid <= flush_tid from the list, release i_ceph_lock, and free
them via ceph_free_cap_flush() outside any lock. When the original
thread reacquires i_ceph_lock and the for-loop macro advances via
cf = list_next_entry(cf, i_list), it dereferences cf->i_list.next
on freed memory.
The race timeline:
__kick_flushing_caps() handle_cap_flush_ack()
----------------------- -----------------------
holds i_ceph_lock <---
iterates to cf (tid=10)
prepares FLUSH message
drops i_ceph_lock <---
__send_cap() ── FLUSH(tid=10)
MDS sends FLUSH_ACK(tid=10)
---> acquires i_ceph_lock
cf->tid(10) <= flush_tid(10),
detaches cf from i_cap_flush_list
drops i_ceph_lock
ceph_free_cap_flush(cf) <- frees it!
acquires i_ceph_lock <---
for-loop advances:
cf = list_next_entry(cf, i_list)
-- UAF on freed cf->i_list.next
The cf was just sent by __kick_flushing_caps itself via __send_cap().
The MDS may respond with FLUSH_ACK quickly enough that
handle_cap_flush_ack() frees cf before __kick_flushing_caps can
finish the iteration.
Fix by converting to a manual while loop: save the next pointer
under i_ceph_lock before dropping it, then use the saved pointer
after reacquiring, so the potentially-freed cf is never accessed again.
Cc: stable@vger.kernel.org
Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[ Upstream commit 50958bb928bad3bdba9e5d1b7ff4bbadcf6951e6 ]
A reader can hang forever in __ceph_get_caps() when the client no
longer holds `FILE_RD`, but local cap state still says that the
capability is already wanted (via `mds_wanted`).
One way to trigger this is through MDS cap revocation. If another
client performs a conflicting operation, the MDS can revoke `FILE_RD`
from the reader; the next read then has to reacquire `FILE_RD`. If
the cap update that should request `FILE_RD` never reaches the MDS
after `cap->mds_wanted` was raised, the reader is left holding only
non-file caps while local `mds_wanted` still includes the file read
caps.
In that state, try_get_cap_refs() sees `need <= mds_wanted` and
returns 0, so __ceph_get_caps() just waits on `i_cap_wq`. If the cap
update that was supposed to request `FILE_RD never reaches the MDS
after `cap->mds_wanted was` raised, no further request is sent and the
waiter can sleep indefinitely until unrelated cap traffic happens to
wake it up.
The ordering issue is that `cap->mds_wanted` is updated in
__prep_cap() before the `CEPH_MSG_CLIENT_CAPS message` is actually
queued for send. That makes one field serve two different meanings at
once: what this client wants, and what the client believes the MDS
already knows it wants.
A proper fix would be to split those states and track whether a cap
update is actually in flight or has been observed by the MDS.
However, simply moving the `cap->mds_wanted assignment` later would
not be sufficient: queueing the message in the messenger does not
guarantee that the MDS processed that specific wanted set, and
reconnect or message loss can still invalidate that assumption.
Fixing that properly would require a larger rework of the cap state
machine.
To allow simpler backports to stable kernels, this patch implements a
simpler workaround:
- stop waiting forever in __ceph_get_caps(); after a bounded wait,
fall back to the renew path
- make ceph_renew_caps() issue a synchronous `OPEN` request whenever
the inode still does not actually hold the wanted caps, instead of
only calling ceph_check_caps()
The extra issued-vs-wanted check in ceph_renew_caps() is necessary
because the previous test only checked whether the inode still had any
real caps at all. That is not enough after revocation: the client can
still hold something like `pLs` and yet be missing `FILE_RD`
completely. In that case, falling back to ceph_check_caps() is not
sufficient, because it still trusts `cap->mds_wanted` and may resend
nothing. By requiring `(issued & wanted) == wanted` before taking the
asynchronous path, the code only uses ceph_check_caps() when the
`wanted caps` are already actually issued. Otherwise, it sends the
synchronous `OPEN` renew.
This preserves the existing asynchronous fast path when the wanted
caps are already issued, avoids changing cap-state semantics, and
fixes the hang by guaranteeing that a stalled waiter eventually
retries through a path that does not rely on the stale `mds_wanted`
state.
[ idryomov: move CEPH_GET_CAPS_WAIT_TIMEOUT from libceph.h to
mds_client.h, formatting ]
Cc: stable@vger.kernel.org
Fixes: 0a454bdd501a ("ceph: reorganize __send_cap for less spinlock abuse")
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Reviewed-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[ Upstream commit 5b602344a49e039e792ce5a8923bcc61412ee134 ]
handle_reply() stores a `ceph_mds_request` pointer in
`current->journal_info` while filling the inode and dentry cache from
an MDS reply.
An allocation in this section can enter direct reclaim and prune
dentries from another filesystem. If this dirties an ext4 inode, ext4
starts a JBD2 transaction. JBD2 interprets the Ceph request in
`current->journal_info` as a journal handle and dereferences the
request's `r_tid` as `h_transaction`, causing a kernel crash, e.g.:
Unable to handle kernel paging request at virtual address 00000000077b4818
[...]
Internal error: Oops: 0000000096000004 [#1] SMP
Modules linked in:
CPU: 6 UID: 0 PID: 2699135 Comm: kworker/6:3 Tainted: G W 6.18.38-i3 #1113 NONE
[...]
Workqueue: ceph-msgr ceph_con_workfn
pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : jbd2__journal_start+0x2c/0x208
lr : __ext4_journal_start_sb+0x100/0x178
[...]
Call trace:
jbd2__journal_start+0x2c/0x208 (P)
__ext4_journal_start_sb+0x100/0x178
ext4_dirty_inode+0x3c/0x90
__mark_inode_dirty+0x58/0x400
iput.part.0+0x2b0/0x370
iput+0x18/0x30
dentry_unlink_inode+0xc0/0x158
__dentry_kill+0x80/0x250
shrink_dentry_list+0x90/0x130
prune_dcache_sb+0x60/0x98
super_cache_scan+0xe8/0x190
do_shrink_slab+0x174/0x388
shrink_slab+0xd8/0x4c0
shrink_node+0x31c/0x908
do_try_to_free_pages+0xd0/0x508
try_to_free_pages+0x11c/0x238
__alloc_frozen_pages_noprof+0x4d0/0xdd0
__folio_alloc_noprof+0x18/0x70
__filemap_get_folio+0x248/0x440
ceph_readdir_prepopulate+0x570/0x9e8
mds_dispatch+0x1424/0x1ba0
ceph_con_process_message+0x74/0xa0
ceph_con_v1_try_read+0x3a0/0x1510
ceph_con_workfn+0x260/0x460
Enter a scoped NOFS allocation context and leave it after clearing
`journal_info`. This prevents filesystem reclaim from recursing into
another filesystem while the field contains Ceph-private data.
Cc: stable@vger.kernel.org
Fixes: 315f24088048 ("ceph: fix security xattr deadlock")
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Reviewed-by: Xiubo Li <xiubo.li@clyso.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>