linux.git/fs/eventpoll.c, branch v2.6.22

epoll: move kfree inside ep_free

2007-05-15T15:54:00+00:00

Move the kfree() call inside the ep_free() function.

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll: fix some comments

2007-05-15T15:54:00+00:00

Fixes some epoll code comments.

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll locks changes and cleanups

2007-05-15T15:53:59+00:00

Changes the rwlock to a spinlock, and drops the use-count variable.
Operations are always bound by the mutex now, so the use-count is no more
needed.  For the same reason, the rwlock can become a simple spinlock.

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

fix epoll single pass code and add wait-exclusive flag

2007-05-15T15:53:59+00:00

Fixes the epoll single pass code.  During the unlocked event delivery (to
userspace) code, the poll callback can re-issue new events, and we must
receive them correctly.  Since we loop in a lockless fashion, we want to be
O(nready), and we don't want to flash on/off the spinlock for every event, we
have the poll callback to use a secondary list to queue events while we're
inside the event delivery loop.  The rw_semaphore has been turned into a
mutex.  This patch also adds the wait-exclusive flag, as suggested by Davi
Arnaut.

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll cleanups: epoll remove static pre-declarations and akpm-ize the code

2007-05-11T15:29:37+00:00

Re-arrange epoll code to avoid static functions pre-declarations, and apply
akpm-filter on it.

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll cleanups: epoll no module

2007-05-11T15:29:37+00:00

Epoll is either compiled it, or not (if EMBEDDED). Remove the module code
and use fs_initcall().

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll: use anonymous inodes

2007-05-11T15:29:37+00:00

Cut out lots of code from epoll, by reusing the anonymous inode source
patch (fs/anon_inodes.c).

Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

Introduce a handy list_first_entry macro

2007-05-08T18:15:11+00:00

There are many places in the kernel where the construction like

   foo = list_entry(head->next, struct foo_struct, list);

are used.
The code might look more descriptive and neat if using the macro

   list_first_entry(head, type, member) \
             list_entry((head)->next, type, member)

Here is the macro itself and the examples of its usage in the generic code.
 If it will turn out to be useful, I can prepare the set of patches to
inject in into arch-specific code, drivers, networking, etc.

Signed-off-by: Pavel Emelianov 
Signed-off-by: Kirill Korotaev 
Cc: Randy Dunlap 
Cc: Andi Kleen 
Cc: Zach Brown 
Cc: Davide Libenzi 
Cc: John McCutchan 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: john stultz 
Cc: Ram Pai 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

header cleaning: don't include smp_lock.h when not used

2007-05-08T18:15:07+00:00

Remove includes of  where it is not used/needed.
Suggested by Al Viro.

Builds cleanly on x86_64, i386, alpha, ia64, powerpc, sparc,
sparc64, and arm (all 59 defconfigs).

Signed-off-by: Randy Dunlap 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

epoll: optimizations and cleanups

2007-05-08T18:15:01+00:00

Epoll is doing multiple passes over the ready set at the moment, because of
the constraints over the f_op->poll() call.  Looking at the code again, I
noticed that we already hold the epoll semaphore in read, and this
(together with other locking conditions that hold while doing an
epoll_wait()) can lead to a smarter way [1] to "ship" events to userspace
(in a single pass).

This is a stress application that can be used to test the new code.  It
spwans multiple thread and call epoll_wait() and epoll_ctl() from many
threads.  Stress tested on my dual Opteron 254 w/out any problems.

http://www.xmailserver.org/totalmess.c

This is not a benchmark, just something that tries to stress and exploit
possible problems with the new code.
Also, I made a stupid micro-benchmark:

http://www.xmailserver.org/epwbench.c

[1] Considering that epoll must be thread-safe, there are five ways we can
    be hit during an epoll_wait() transfer loop (ep_send_events()):

    1) The epoll fd going away and calling ep_free
       This just can't happen, since we did an fget() in sys_epoll_wait

    2) An epoll_ctl(EPOLL_CTL_DEL)
       This can't happen because epoll_ctl() gets ep->sem in write, and
       we're holding it in read during ep_send_events()

    3) An fd stored inside the epoll fd going away
       This can't happen because in eventpoll_release_file() we get
       ep->sem in write, and we're holding it in read during
       ep_send_events()

    4) Another epoll_wait() happening on another thread
       They both can be inside ep_send_events() at the same time, we get
       (splice) the ready-list under the spinlock, so each one will get
       its own ready list. Note that an fd cannot be at the same time
       inside more than one ready list, because ep_poll_callback() will
       not re-queue it if it sees it already linked:

       if (ep_is_linked(&epi->rdllink))
                goto is_linked;

       Another case that can happen, is two concurrent epoll_wait(),
       coming in with a userspace event buffer of size, say, ten.
       Suppose there are 50 event ready in the list. The first
       epoll_wait() will "steal" the whole list, while the second, seeing
       no events, will go to sleep. But at the end of ep_send_events() in
       the first epoll_wait(), we will re-inject surplus ready fds, and we
       will trigger the proper wake_up to the second epoll_wait().

    5) ep_poll_callback() hitting us asyncronously
       This is the tricky part. As I said above, the ep_is_linked() test
       done inside ep_poll_callback(), will guarantee us that until the
       item will result linked to a list, ep_poll_callback() will not try
       to re-queue it again (read, write data on any of its members). When
       we do a list_del() in ep_send_events(), the item will still satisfy
       the ep_is_linked() test (whatever data is written in prev/next,
       it'll never be its own pointer), so ep_poll_callback() will still
       leave us alone. It's only after the eventual smp_mb()+INIT_LIST_HEAD(&epi->rdllink)
       that it'll become visible to ep_poll_callback(), but at the point
       we're already past it.

[akpm@osdl.org: 80 cols]
Signed-off-by: Davide Libenzi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds