linux.git/fs/eventpoll.c, branch v7.2 Linux kernel source tree eventpoll: pin files while checking reverse paths 2026-07-23T10:48:22+00:00 Guidong Han 2045gemini@gmail.com 2026-07-18T10:44:06+00:00 8b7e8245e2293078f657521236ac92c045552e5a Commit 319c15174757 ("epoll: take epitem list out of struct file") intentionally removed temporary file references from the reverse path check list. At the time, both epitems and their files were freed after an RCU grace period, so unlist_file() could obtain file->f_lock through an epitem while clear_tfile_check_list() held rcu_read_lock(). Commit 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU") made struct file SLAB_TYPESAFE_BY_RCU and removed its RCU-delayed freeing. RCU still protects the epitem, but no longer keeps the referenced file from being freed and reused. A concurrent close can therefore make unlist_file() lock or unlock f_lock in a recycled file object. This violates the documented SLAB_TYPESAFE_BY_RCU rule requiring a reference before acquiring an object's lock. The race was reproduced, causing a wild unlock of f_lock in a recycled file and breaking its mutual exclusion. Add ->file to epitems_head to remember the pinned file independently of ->epitems. A concurrent EPOLL_CTL_DEL can empty ->epitems before the head is unlisted, leaving no epi->ffd.file from which to drop the reference. In list_file(), acquire the reference before adding the head to the check list. The caller either owns a reference or holds the ep->mtx for the epitem leading to the file. In the latter case, file_ref_get() can fail after the last reference is dropped, but eventpoll_release_file() must acquire the same mutex before the file can be freed. The dying leaf can be skipped because removing links cannot increase the reverse path count. In unlist_file(), epnested_mutex excludes another list_file() or unlist_file(), while head->next prevents a concurrent EPOLL_CTL_DEL from freeing the head. Save head->file locally, clear it with head->next under f_lock, and drop the reference after the RCU-protected operation. Christian Brauner <brauner@kernel.org> quotes: > SLAB_TYPESAFE_BY_RCU allows a slab slot to be reused while an RCU reader > still holds its old address. Once that address contains a new live > struct file, KASAN sees valid, unpoisoned memory and cannot distinguish > the stale object identity. CONFIG_DEBUG_SPINLOCK exposes the failure > instead. > > The failing interleaving is: > > CPU0: nested EPOLL_CTL_ADD CPU1: close/open churn > ------------------------------------ --------------------------------- > p = hlist_first_rcu(&head->epitems) > epi = container_of(p, ...) > close(victim) > __fput() > eventpoll_release_file() > file_free(victim) > // the slot is free; f_lock remains > spin_lock(&epi->ffd.file->f_lock) > open() reuses the slot as new_file > spin_lock_init(&new_file->f_lock) > spin_unlock(&epi->ffd.file->f_lock) // wild unlock of new_file's lock > > CONFIG_DEBUG_SPINLOCK reports: > > BUG: spinlock already unlocked on CPU#0, poc_unlist/150 > lock: 0xffff8880067fb200, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1 > CPU: 0 UID: 1000 PID: 150 Comm: poc_unlist Not tainted 7.2.0-rc3-dirty #22 PREEMPTLAZY > Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 > Call Trace: > <TASK> > dump_stack_lvl+0x64/0x80 > do_raw_spin_unlock+0x75/0xb0 > _raw_spin_unlock+0xe/0x30 > clear_tfile_check_list+0x88/0xe0 > do_epoll_ctl_file+0x519/0xcf0 > ? __pfx_ep_ptable_queue_proc+0x10/0x10 > do_epoll_ctl+0x8f/0x100 > __x64_sys_epoll_ctl+0x6f/0xa0 > do_syscall_64+0xdc/0x520 > ? srso_alias_return_thunk+0x5/0xfbef5 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x42034e > Code: 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007a657ff3c198 EFLAGS: 00000202 ORIG_RAX: 00000000000000e9 > RAX: ffffffffffffffda RBX: 00007a657ff3ccdc RCX: 000000000042034e > RDX: 0000000000000003 RSI: 0000000000000001 RDI: 0000000000000004 > RBP: 00007a657ff3c2f0 R08: 0000000000000000 R09: 00007a657ff3c6c0 > R10: 00007a657ff3c1a4 R11: 0000000000000202 R12: 00007a657ff3c6c0 > R13: ffffffffffffffb8 R14: 000000000000000d R15: 00007fffb7de0210 > </TASK> > ------------[ cut here ]------------ > > unlist_file() does not appear as a separate frame because it was inlined > into clear_tfile_check_list(). This report was obtained with mdelay() > instrumentation immediately before spin_lock() and spin_unlock() in > unlist_file() to widen the two race windows. > > More importantly, this is a wild unlock. The stale unlock can target > f_lock of a different live file and invalidate mutual exclusion for > state protected by that lock. Turning this into a reliable exploit > would require precise scheduling and same-slot reuse and is likely > difficult, but the primitive is potentially exploitable. Reported-by: Qi Tang <tpluszz77@gmail.com> Reported-by: Junxi Qian <qjx1298677004@gmail.com> Fixes: 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU") Cc: stable@vger.kernel.org Signed-off-by: Guidong Han <2045gemini@gmail.com> Link: https://patch.msgid.link/20260718104406.27897-1-2045gemini@gmail.com Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> 
Commit 319c15174757 ("epoll: take epitem list out of struct file")
intentionally removed temporary file references from the reverse path
check list. At the time, both epitems and their files were freed after
an RCU grace period, so unlist_file() could obtain file->f_lock through
an epitem while clear_tfile_check_list() held rcu_read_lock().

Commit 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU") made
struct file SLAB_TYPESAFE_BY_RCU and removed its RCU-delayed freeing.
RCU still protects the epitem, but no longer keeps the referenced file
from being freed and reused. A concurrent close can therefore make
unlist_file() lock or unlock f_lock in a recycled file object.

This violates the documented SLAB_TYPESAFE_BY_RCU rule requiring a
reference before acquiring an object's lock. The race was reproduced,
causing a wild unlock of f_lock in a recycled file and breaking its
mutual exclusion.

Add ->file to epitems_head to remember the pinned file independently of
->epitems. A concurrent EPOLL_CTL_DEL can empty ->epitems before the head
is unlisted, leaving no epi->ffd.file from which to drop the reference.

In list_file(), acquire the reference before adding the head to the
check list. The caller either owns a reference or holds the ep->mtx for
the epitem leading to the file. In the latter case, file_ref_get() can
fail after the last reference is dropped, but eventpoll_release_file()
must acquire the same mutex before the file can be freed. The dying leaf
can be skipped because removing links cannot increase the reverse path
count.

In unlist_file(), epnested_mutex excludes another list_file() or
unlist_file(), while head->next prevents a concurrent EPOLL_CTL_DEL from
freeing the head. Save head->file locally, clear it with head->next
under f_lock, and drop the reference after the RCU-protected operation.

Christian Brauner <brauner@kernel.org> quotes:

> SLAB_TYPESAFE_BY_RCU allows a slab slot to be reused while an RCU reader
> still holds its old address. Once that address contains a new live
> struct file, KASAN sees valid, unpoisoned memory and cannot distinguish
> the stale object identity. CONFIG_DEBUG_SPINLOCK exposes the failure
> instead.
>
> The failing interleaving is:
>
> CPU0: nested EPOLL_CTL_ADD             CPU1: close/open churn
> ------------------------------------   ---------------------------------
> p = hlist_first_rcu(&head->epitems)
> epi = container_of(p, ...)
>                                        close(victim)
>                                          __fput()
>                                            eventpoll_release_file()
>                                            file_free(victim)
>                                        // the slot is free; f_lock remains
> spin_lock(&epi->ffd.file->f_lock)
>                                        open() reuses the slot as new_file
>                                          spin_lock_init(&new_file->f_lock)
> spin_unlock(&epi->ffd.file->f_lock)     // wild unlock of new_file's lock
>
> CONFIG_DEBUG_SPINLOCK reports:
>
> BUG: spinlock already unlocked on CPU#0, poc_unlist/150
>  lock: 0xffff8880067fb200, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
> CPU: 0 UID: 1000 PID: 150 Comm: poc_unlist Not tainted 7.2.0-rc3-dirty #22 PREEMPTLAZY
> Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> Call Trace:
>  <TASK>
>  dump_stack_lvl+0x64/0x80
>  do_raw_spin_unlock+0x75/0xb0
>  _raw_spin_unlock+0xe/0x30
>  clear_tfile_check_list+0x88/0xe0
>  do_epoll_ctl_file+0x519/0xcf0
>  ? __pfx_ep_ptable_queue_proc+0x10/0x10
>  do_epoll_ctl+0x8f/0x100
>  __x64_sys_epoll_ctl+0x6f/0xa0
>  do_syscall_64+0xdc/0x520
>  ? srso_alias_return_thunk+0x5/0xfbef5
>  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> RIP: 0033:0x42034e
> Code: 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007a657ff3c198 EFLAGS: 00000202 ORIG_RAX: 00000000000000e9
> RAX: ffffffffffffffda RBX: 00007a657ff3ccdc RCX: 000000000042034e
> RDX: 0000000000000003 RSI: 0000000000000001 RDI: 0000000000000004
> RBP: 00007a657ff3c2f0 R08: 0000000000000000 R09: 00007a657ff3c6c0
> R10: 00007a657ff3c1a4 R11: 0000000000000202 R12: 00007a657ff3c6c0
> R13: ffffffffffffffb8 R14: 000000000000000d R15: 00007fffb7de0210
>  </TASK>
> ------------[ cut here ]------------
>
> unlist_file() does not appear as a separate frame because it was inlined
> into clear_tfile_check_list(). This report was obtained with mdelay()
> instrumentation immediately before spin_lock() and spin_unlock() in
> unlist_file() to widen the two race windows.
>
> More importantly, this is a wild unlock. The stale unlock can target
> f_lock of a different live file and invalidate mutual exclusion for
> state protected by that lock. Turning this into a reliable exploit
> would require precise scheduling and same-slot reuse and is likely
> difficult, but the primitive is potentially exploitable.

Reported-by: Qi Tang <tpluszz77@gmail.com>
Reported-by: Junxi Qian <qjx1298677004@gmail.com>
Fixes: 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU")
Cc: stable@vger.kernel.org
Signed-off-by: Guidong Han <2045gemini@gmail.com>
Link: https://patch.msgid.link/20260718104406.27897-1-2045gemini@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
eventpoll: restore EP_UNACTIVE_PTR sentinel for ctx->tfile_check_list 2026-06-04T11:53:50+00:00 Zhan Wei zhanwei919@gmail.com 2026-05-29T14:25:33+00:00 a1e9718b406bc4e6d0c63c7b999d06febbdc4091 Commit e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") moved tfile_check_list from a file-scope global into the stack-allocated struct ep_ctl_ctx, and in doing so replaced the EP_UNACTIVE_PTR sentinel with NULL on the grounds that "NULL is the obvious 'empty' value and zero-init handles it for free", describing the change as "No functional change". It is not. epitems_head->next is overloaded with two roles: 1. the "next" pointer that threads a head onto ctx->tfile_check_list; 2. a membership flag: ep_remove_file() uses !smp_load_acquire(&v->next) to mean "this head is not on any pending ctx->tfile_check_list and is therefore safe to free". Before that change the EP_UNACTIVE_PTR sentinel kept the two roles disjoint: a head on the list always had a non-NULL ->next (another head, or the sentinel at the tail), so ->next == NULL was equivalent to "never listed". With the sentinel gone the list is NULL-terminated, so the tail head's ->next is NULL as well. ep_remove_file()'s gate can no longer distinguish "never listed" from "listed at the tail", and misfires on the tail head. The reader (reverse_path_check_proc) holds epnested_mutex + rcu_read_lock; the freer (ep_remove_file) holds ep->mtx + file->f_lock. The two sides share no mutex -- the sentinel was the invariant the gate relied on to know it could skip the read side. With it gone, ep_remove_file() frees the tail head while reverse_path_check_proc() is still walking it, producing the slab-use-after-free read. The syzbot reproducer hits this within seconds on a multi-CPU VM. Restore the sentinel: initialize ctx.tfile_check_list to EP_UNACTIVE_PTR in do_epoll_ctl_file(), and terminate the walk on "!= EP_UNACTIVE_PTR" in reverse_path_check() and clear_tfile_check_list(). The tail head's ->next becomes the sentinel again rather than NULL, so ep_remove_file()'s gate regains its exclusivity and stops misfiring on the tail. ep_remove_file() itself is unchanged. This restores the invariant the file-scope tfile_check_list relied on before that change while preserving the ctx packaging it introduced. Reported-by: syzbot+e70e1b6cba8714543f7c@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=e70e1b6cba8714543f7c Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Suggested-by: Christian Brauner <brauner@kernel.org> Link: https://lore.kernel.org/all/20260528-rotwild-summt-kuhhandel-7276ef4c33b7@brauner.io/ Signed-off-by: Zhan Wei <zhanwei919@gmail.com> Link: https://patch.msgid.link/20260529142533.23696-1-zhanwei919@gmail.com Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> 
Commit e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into
struct ep_ctl_ctx") moved tfile_check_list from a file-scope global into
the stack-allocated struct ep_ctl_ctx, and in doing so replaced the
EP_UNACTIVE_PTR sentinel with NULL on the grounds that "NULL is the
obvious 'empty' value and zero-init handles it for free", describing the
change as "No functional change". It is not.

epitems_head->next is overloaded with two roles:

  1. the "next" pointer that threads a head onto ctx->tfile_check_list;
  2. a membership flag: ep_remove_file() uses
     !smp_load_acquire(&v->next) to mean "this head is not on any
     pending ctx->tfile_check_list and is therefore safe to free".

Before that change the EP_UNACTIVE_PTR sentinel kept the two roles
disjoint: a head on the list always had a non-NULL ->next (another head,
or the sentinel at the tail), so ->next == NULL was equivalent to "never
listed". With the sentinel gone the list is NULL-terminated, so the tail
head's ->next is NULL as well. ep_remove_file()'s gate can no longer
distinguish "never listed" from "listed at the tail", and misfires on
the tail head.

The reader (reverse_path_check_proc) holds epnested_mutex +
rcu_read_lock; the freer (ep_remove_file) holds ep->mtx + file->f_lock.
The two sides share no mutex -- the sentinel was the invariant the gate
relied on to know it could skip the read side. With it gone,
ep_remove_file() frees the tail head while reverse_path_check_proc() is
still walking it, producing the slab-use-after-free read. The syzbot
reproducer hits this within seconds on a multi-CPU VM.

Restore the sentinel: initialize ctx.tfile_check_list to EP_UNACTIVE_PTR
in do_epoll_ctl_file(), and terminate the walk on "!= EP_UNACTIVE_PTR"
in reverse_path_check() and clear_tfile_check_list(). The tail head's
->next becomes the sentinel again rather than NULL, so
ep_remove_file()'s gate regains its exclusivity and stops misfiring on
the tail. ep_remove_file() itself is unchanged.

This restores the invariant the file-scope tfile_check_list relied on
before that change while preserving the ctx packaging it introduced.

Reported-by: syzbot+e70e1b6cba8714543f7c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=e70e1b6cba8714543f7c
Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx")
Suggested-by: Christian Brauner <brauner@kernel.org>
Link: https://lore.kernel.org/all/20260528-rotwild-summt-kuhhandel-7276ef4c33b7@brauner.io/
Signed-off-by: Zhan Wei <zhanwei919@gmail.com>
Link: https://patch.msgid.link/20260529142533.23696-1-zhanwei919@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
eventpoll: Fix epoll_wait() report false negative 2026-06-04T08:25:10+00:00 Nam Cao namcao@linutronix.de 2026-06-02T17:51:46+00:00 0c4aefe3c2d0f272a2ad73699a12d4446ffdbe7b ep_events_available() checks for available events by looking at ep->rdllist and ep_is_scanning(). However, this is done without a lock and can report false negative if ep_start_scan() or ep_done_scan() are executed by another task concurrently. For example: _________________________________________________________________________ |ep_start_scan() | list_splice_init(&ep->rdllist, ...) ep_events_available() | !list_empty_careful(&ep->rdllist)| || ep_is_scanning(ep) | | ep_enter_scan(ep) ___________________________________|_____________________________________ Another example: _________________________________________________________________________ ep_events_available() | |ep_start_scan() | list_splice_init(&ep->rdllist, ...) | ep_enter_scan(ep) !list_empty_careful(&ep->rdllist)| |ep_done_scan() | ep_exit_scan(ep) | list_splice(..., &ep->rdllist) || ep_is_scanning(ep) | ___________________________________|_____________________________________ In the above examples, ep_events_available() sees no event despite events being available. In case epoll_wait() is called with timeout=0, epoll_wait() will wrongly return "no event" to user. Introduce a sequence lock to resolve this issue. Measuring the time consumption of 10 million loop iterations doing epoll_wait(), the following performance drop is observed: timeout #event before after diff 0ms 0 3727ms 3974ms +6.6% 0ms 1 8099ms 9134ms +13% 1ms 1 13525ms 13586ms +0.45% Considering the use case of epoll_wait() (wait for events, do something with the events, repeat), it should only contribute to a small portion of user's CPU consumption. Therefore this performance drop is not alarming. Fixes: c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()") Suggested-by: Mateusz Guzik <mjguzik@gmail.com> Signed-off-by: Nam Cao <namcao@linutronix.de> Link: https://patch.msgid.link/4363cd8e34a21d4f0d257be1b33e84dc25030fdf.1780422138.git.namcao@linutronix.de Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> 
ep_events_available() checks for available events by looking at
ep->rdllist and ep_is_scanning(). However, this is done without a lock
and can report false negative if ep_start_scan() or ep_done_scan() are
executed by another task concurrently. For example:
_________________________________________________________________________
                                   |ep_start_scan()
                                   |  list_splice_init(&ep->rdllist, ...)
ep_events_available()              |
  !list_empty_careful(&ep->rdllist)|
  || ep_is_scanning(ep)            |
	                           |  ep_enter_scan(ep)
___________________________________|_____________________________________

Another example:
_________________________________________________________________________
ep_events_available()              |
                                   |ep_start_scan()
                                   |  list_splice_init(&ep->rdllist, ...)
	                           |  ep_enter_scan(ep)
  !list_empty_careful(&ep->rdllist)|
                                   |ep_done_scan()
                                   |  ep_exit_scan(ep)
                                   |  list_splice(..., &ep->rdllist)
  || ep_is_scanning(ep)            |
___________________________________|_____________________________________

In the above examples, ep_events_available() sees no event despite
events being available. In case epoll_wait() is called with timeout=0,
epoll_wait() will wrongly return "no event" to user.

Introduce a sequence lock to resolve this issue.

Measuring the time consumption of 10 million loop iterations doing
epoll_wait(), the following performance drop is observed:

   timeout  #event  before    after    diff
     0ms      0     3727ms   3974ms   +6.6%
     0ms      1     8099ms   9134ms    +13%
     1ms      1    13525ms  13586ms  +0.45%

Considering the use case of epoll_wait() (wait for events, do something
with the events, repeat), it should only contribute to a small portion of
user's CPU consumption. Therefore this performance drop is not alarming.

Fixes: c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()")
Suggested-by: Mateusz Guzik <mjguzik@gmail.com>
Signed-off-by: Nam Cao <namcao@linutronix.de>
Link: https://patch.msgid.link/4363cd8e34a21d4f0d257be1b33e84dc25030fdf.1780422138.git.namcao@linutronix.de
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
eventpoll: add missing kernel-doc for @ctx function parameters 2026-05-22T12:32:06+00:00 Randy Dunlap rdunlap@infradead.org 2026-05-19T04:23:14+00:00 70a03a385de2b8f0fa54dbc70bdc3ed176853d1c Add the missing kernel-doc comments to prevent kernel-doc build warnings while building the documentation. WARNING: fs/eventpoll.c:1684 function parameter 'ctx' not described in 'reverse_path_check' WARNING: fs/eventpoll.c:2349 function parameter 'ctx' not described in 'ep_loop_check_proc' Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Signed-off-by: Randy Dunlap <rdunlap@infradead.org> Link: https://patch.msgid.link/20260519042314.124041-1-rdunlap@infradead.org Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> 
Add the missing kernel-doc comments to prevent kernel-doc build
warnings while building the documentation.

WARNING: fs/eventpoll.c:1684 function parameter 'ctx' not described in 'reverse_path_check'
WARNING: fs/eventpoll.c:2349 function parameter 'ctx' not described in 'ep_loop_check_proc'

Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx")
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Link: https://patch.msgid.link/20260519042314.124041-1-rdunlap@infradead.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Merge patch series "io_uring related epoll cleanups" 2026-05-15T15:41:05+00:00 Christian Brauner brauner@kernel.org 2026-05-15T15:41:05+00:00 6ece1a31c58c8c8293ecbbe79d7f92d52e1b0022 Jens Axboe <axboe@kernel.dk> says: One of the nastier things about epoll is how it allows nesting contexts inside each other, leading to the necessity of loop detection and the issues that have come with that. I don't believe there's any reason to support nesting on the io_uring side, in fact IORING_OP_EPOLL_CTL is a historical mistake, imho. But let's at least try and contain the damage and disallow nested contexts from our side. Christian Brauner <brauner@kernel.org> says: Bring in the eventpoll specific io_uring changes together with the eventpoll cleanup I did this cycle. The io_uring changes can go on top of both through the block tree. * patches from https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk: eventpoll: rename struct epoll_filefd to epoll_key eventpoll: add file based control interface eventpoll: export is_file_epoll() eventpoll: pass struct epoll_filefd through ep_find() and ep_insert() Link: https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> 
Jens Axboe <axboe@kernel.dk> says:

One of the nastier things about epoll is how it allows nesting contexts
inside each other, leading to the necessity of loop detection and the
issues that have come with that.

I don't believe there's any reason to support nesting on the io_uring
side, in fact IORING_OP_EPOLL_CTL is a historical mistake, imho. But
let's at least try and contain the damage and disallow nested contexts
from our side.

Christian Brauner <brauner@kernel.org> says:

Bring in the eventpoll specific io_uring changes together with the
eventpoll cleanup I did this cycle. The io_uring changes can go on top
of both through the block tree.

* patches from https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk:
  eventpoll: rename struct epoll_filefd to epoll_key
  eventpoll: add file based control interface
  eventpoll: export is_file_epoll()
  eventpoll: pass struct epoll_filefd through ep_find() and ep_insert()

Link: https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
eventpoll: rename struct epoll_filefd to epoll_key 2026-05-15T15:34:26+00:00 Jens Axboe axboe@kernel.dk 2026-05-14T14:07:20+00:00 463aba8090738bcd956297f7341b6874e76ae664 This more accurately describes what purpose this structure serves, as a lookup key. Suggested-by: Christian Brauner <brauner@kernel.org> Signed-off-by: Jens Axboe <axboe@kernel.dk> Link: https://patch.msgid.link/20260514140817.623026-5-axboe@kernel.dk Signed-off-by: Christian Brauner <brauner@kernel.org> 
This more accurately describes what purpose this structure serves, as
a lookup key.

Suggested-by: Christian Brauner <brauner@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Link: https://patch.msgid.link/20260514140817.623026-5-axboe@kernel.dk
Signed-off-by: Christian Brauner <brauner@kernel.org>
eventpoll: add file based control interface 2026-05-15T15:34:26+00:00 Jens Axboe axboe@kernel.dk 2026-05-14T14:07:19+00:00 0995baf261ce8fc141768911d97ba76e854e26cf Add do_epoll_ctl_file(), which takes a pre-resolved epoll file and a struct epoll_filefd for the target rather than two integer file descriptors. do_epoll_ctl() remains as a thin wrapper. In preparation for using the file based interface from io_uring. Signed-off-by: Jens Axboe <axboe@kernel.dk> Link: https://patch.msgid.link/20260514140817.623026-4-axboe@kernel.dk Signed-off-by: Christian Brauner <brauner@kernel.org> 
Add do_epoll_ctl_file(), which takes a pre-resolved epoll file and a
struct epoll_filefd for the target rather than two integer file
descriptors. do_epoll_ctl() remains as a thin wrapper.

In preparation for using the file based interface from io_uring.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Link: https://patch.msgid.link/20260514140817.623026-4-axboe@kernel.dk
Signed-off-by: Christian Brauner <brauner@kernel.org>
eventpoll: export is_file_epoll() 2026-05-15T15:34:26+00:00 Jens Axboe axboe@kernel.dk 2026-05-14T14:07:18+00:00 5de2759f2b7c925f187e552cae47775acd5f4b40 Make is_file_epoll() available outside of epoll. This is in preparation from using it from io_uring. Signed-off-by: Jens Axboe <axboe@kernel.dk> Link: https://patch.msgid.link/20260514140817.623026-3-axboe@kernel.dk Reviewed-by: Christian Brauner <brauner@kernel.org> Signed-off-by: Christian Brauner <brauner@kernel.org> 
Make is_file_epoll() available outside of epoll. This is in preparation
from using it from io_uring.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Link: https://patch.msgid.link/20260514140817.623026-3-axboe@kernel.dk
Reviewed-by: Christian Brauner <brauner@kernel.org>
Signed-off-by: Christian Brauner <brauner@kernel.org>
eventpoll: pass struct epoll_filefd through ep_find() and ep_insert() 2026-05-15T15:34:26+00:00 Jens Axboe axboe@kernel.dk 2026-05-14T14:07:17+00:00 9caef5b2a1fc22c2e24159a3d0fd74a47aed94ab Have ep_find() and ep_insert() take a struct epoll_filefd rather than a file/fd tuple. Kill off ep_set_ffd() as it's now no longer needed. No functional change. This is a prep patch for adding a file based do_epoll_ctl() variant. Signed-off-by: Jens Axboe <axboe@kernel.dk> Link: https://patch.msgid.link/20260514140817.623026-2-axboe@kernel.dk Reviewed-by: Christian Brauner <brauner@kernel.org> Signed-off-by: Christian Brauner <brauner@kernel.org> 
Have ep_find() and ep_insert() take a struct epoll_filefd rather
than a file/fd tuple. Kill off ep_set_ffd() as it's now no longer
needed.

No functional change. This is a prep patch for adding a file based
do_epoll_ctl() variant.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Link: https://patch.msgid.link/20260514140817.623026-2-axboe@kernel.dk
Reviewed-by: Christian Brauner <brauner@kernel.org>
Signed-off-by: Christian Brauner <brauner@kernel.org>
eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx 2026-04-28T15:27:29+00:00 Christian Brauner brauner@kernel.org 2026-04-24T13:46:48+00:00 e09c77d940034e3f0c4ae1c18cc655ea9ec2cc41 Three globals were shared between the loop check and the path check paths: tfile_check_list (chain of epitems_head to walk afterwards), path_count[] (per-depth wakeup-path tally) and inserting_into (cycle-detection sentinel). All three are scratch state used only during a single EPOLL_CTL_ADD full_check, yet they sit at file scope and rely on epnested_mutex for exclusion. The area has had three bugs in the last year -- CVE-2025-38349, f2e467a48287 ("eventpoll: Fix semi-unbounded recursion"), and fdcfce93073d ("eventpoll: Fix integer overflow in ep_loop_check_proc()") -- all rooted in the shared-mutable-global pattern being hard to reason about. Collect the three into a stack-allocated struct ep_ctl_ctx: struct ep_ctl_ctx { struct eventpoll *inserting_into; struct epitems_head *tfile_check_list; int path_count[PATH_ARR_SIZE]; }; do_epoll_ctl() zero-initializes one on its stack and plumbs it through ep_ctl_lock() / ep_ctl_unlock() / ep_insert() / ep_register_epitem() / list_file() / ep_loop_check() / ep_loop_check_proc() / reverse_path_check() / reverse_path_check_proc() / path_count_inc() / path_count_init() / clear_tfile_check_list(). Non-nested inserts leave the ctx zeroed and skip the machinery entirely. With the scratch state in ctx: - tfile_check_list no longer has an EP_UNACTIVE_PTR sentinel -- NULL is the obvious "empty" value and the zero-init handles it for free; - path_count[] is no longer an array global that could be touched in unexpected orderings; - inserting_into is scoped to the exact call that set it. loop_check_gen stays as a file-scope monotonic counter, because the stamp left on ep->gen by a completed walk must not equal the stamp of a future walk -- something a stack-local value cannot guarantee across calls. It remains protected by epnested_mutex for the bump and read lockless for the "do we need a full check" trigger in ep_ctl_lock(). Every bail-out that existed before (the ELOOP on cycle, the path limit check, the unbounded-recursion cap, the +1 overflow guard) is preserved verbatim; only the data they operate on moved from file scope to the stack ctx. No functional change. Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-17-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner <brauner@kernel.org> 
Three globals were shared between the loop check and the path check
paths: tfile_check_list (chain of epitems_head to walk afterwards),
path_count[] (per-depth wakeup-path tally) and inserting_into
(cycle-detection sentinel). All three are scratch state used only
during a single EPOLL_CTL_ADD full_check, yet they sit at file
scope and rely on epnested_mutex for exclusion.

The area has had three bugs in the last year -- CVE-2025-38349,
f2e467a48287 ("eventpoll: Fix semi-unbounded recursion"), and
fdcfce93073d ("eventpoll: Fix integer overflow in
ep_loop_check_proc()") -- all rooted in the shared-mutable-global
pattern being hard to reason about.

Collect the three into a stack-allocated struct ep_ctl_ctx:

    struct ep_ctl_ctx {
        struct eventpoll   *inserting_into;
        struct epitems_head *tfile_check_list;
        int                 path_count[PATH_ARR_SIZE];
    };

do_epoll_ctl() zero-initializes one on its stack and plumbs it
through ep_ctl_lock() / ep_ctl_unlock() / ep_insert() /
ep_register_epitem() / list_file() / ep_loop_check() /
ep_loop_check_proc() / reverse_path_check() /
reverse_path_check_proc() / path_count_inc() / path_count_init() /
clear_tfile_check_list(). Non-nested inserts leave the ctx zeroed
and skip the machinery entirely.

With the scratch state in ctx:
  - tfile_check_list no longer has an EP_UNACTIVE_PTR sentinel --
    NULL is the obvious "empty" value and the zero-init handles it
    for free;
  - path_count[] is no longer an array global that could be touched
    in unexpected orderings;
  - inserting_into is scoped to the exact call that set it.

loop_check_gen stays as a file-scope monotonic counter, because the
stamp left on ep->gen by a completed walk must not equal the stamp
of a future walk -- something a stack-local value cannot guarantee
across calls. It remains protected by epnested_mutex for the bump
and read lockless for the "do we need a full check" trigger in
ep_ctl_lock().

Every bail-out that existed before (the ELOOP on cycle, the path
limit check, the unbounded-recursion cap, the +1 overflow guard) is
preserved verbatim; only the data they operate on moved from file
scope to the stack ctx.

No functional change.

Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-17-249ed00a20f3@kernel.org
Signed-off-by: Christian Brauner <brauner@kernel.org>