linux.git/lib/rwsem-spinlock.c, branch master

locking: Move the rwsem code to kernel/locking/

2013-11-06T08:24:18+00:00

Notably: changed lib/rwsem* targets from lib- to obj-, no idea about
the ramifications of that.

Suggested-by: Ingo Molnar 
Signed-off-by: Peter Zijlstra 
Link: http://lkml.kernel.org/n/tip-g0kynfh5feriwc6p3h6kpbw6@git.kernel.org
Signed-off-by: Ingo Molnar

rwsem: simplify __rwsem_do_wake

2013-05-07T14:20:16+00:00

This is mostly for cleanup value:

- We don't need several gotos to handle the case where the first
  waiter is a writer. Two simple tests will do (and generate very
  similar code).

- In the remainder of the function, we know the first waiter is a reader,
  so we don't have to double check that. We can use do..while loops
  to iterate over the readers to wake (generates slightly better code).

Signed-off-by: Michel Lespinasse 
Reviewed-by: Peter Hurley 
Acked-by: Davidlohr Bueso 
Signed-off-by: Linus Torvalds

rwsem: make the waiter type an enumeration rather than a bitmask

2013-05-07T14:20:15+00:00

We are not planning to add some new waiter flags, so we can convert the
waiter type into an enumeration.

Background: David Howells suggested I do this back when I tried adding
a new waiter type for unfair readers. However, I believe the cleanup
applies regardless of that use case.

Signed-off-by: Michel Lespinasse 
Reviewed-by: Rik van Riel 
Reviewed-by: Peter Hurley 
Acked-by: Davidlohr Bueso 
Signed-off-by: Linus Torvalds

rwsem-spinlock: Implement writer lock-stealing for better scalability

2013-02-19T07:43:39+00:00

We (Linux Kernel Performance project) found a regression
introduced by commit:

  5a505085f043 mm/rmap: Convert the struct anon_vma::mutex to an rwsem

which converted all anon_vma::mutex locks rwsem write locks.

The semantics are the same, but the behavioral difference is
quite huge in some cases. After investigating it we found the
root cause: mutexes support lock stealing while rwsems don't.

Here is the link for the detailed regression report:

  https://lkml.org/lkml/2013/1/29/84

Ingo suggested adding write lock stealing to rwsems:

    "I think we should allow lock-steal between rwsem writers - that
     will not hurt fairness as most rwsem fairness concerns relate to
     reader vs. writer fairness"

And here is the rwsem-spinlock version.

With this patch, we got a double performance increase in one
test box with following aim7 workfile:

    FILESIZE: 1M
    POOLSIZE: 10M
    10 fork_test

 /usr/bin/time output w/o patch                       /usr/bin/time_output with patch
 -- Percent of CPU this job got: 369%                 Percent of CPU this job got: 537%
 Voluntary context switches: 640595016                Voluntary context switches: 157915561

We got a 45% increase in CPU usage and saved about 3/4 voluntary context switches.

Reported-by: LKP project 
Suggested-by: Ingo Molnar 
Signed-off-by: Yuanhan Liu 
Cc: Alex Shi 
Cc: David Howells 
Cc: Michel Lespinasse 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: Peter Zijlstra 
Cc: Anton Blanchard 
Cc: Arjan van de Ven 
Cc: paul.gortmaker@windriver.com
Link: http://lkml.kernel.org/r/1359716356-23865-1-git-send-email-yuanhan.liu@linux.intel.com
Signed-off-by: Ingo Molnar

lib: reduce the use of module.h wherever possible

2012-03-07T20:04:04+00:00

For files only using THIS_MODULE and/or EXPORT_SYMBOL, map
them onto including export.h -- or if the file isn't even
using those, then just delete the include.  Fix up any implicit
include dependencies that were being masked by module.h along
the way.

Signed-off-by: Paul Gortmaker

locking, rwsem: Annotate inner lock as raw

2011-09-13T09:11:59+00:00

There is no reason to allow the lock protecting rwsems (the
ownerless variant) to be preemptible on -rt. Convert it to raw.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar

rwsem generic spinlock: use IRQ save/restore spinlocks

2010-04-07T23:15:05+00:00

rwsems can be used with IRQs disabled, particularily in early boot
before IRQs are enabled.  Currently the spin_unlock_irq() usage in the
slow-patch will unconditionally enable interrupts and cause problems
since interrupts are not yet initialized or enabled.

This patch uses save/restore versions of IRQ spinlocks in the slowpath
to ensure interrupts are not unintentionally disabled.

Signed-off-by: Kevin Hilman 
Signed-off-by: Linus Torvalds

rwsem: fix rwsem_is_locked() bugs

2009-12-15T16:53:26+00:00

rwsem_is_locked() tests ->activity without locks, so we should always keep
->activity consistent.  However, the code in __rwsem_do_wake() breaks this
rule, it updates ->activity after _all_ readers waken up, this may give
some reader a wrong ->activity value, thus cause rwsem_is_locked() behaves
wrong.

Quote from Andrew:

"
- we have one or more processes sleeping in down_read(), waiting for access.

- we wake one or more processes up without altering ->activity

- they start to run and they do rwsem_is_locked().  This incorrectly
  returns "false", because the waker process is still crunching away in
  __rwsem_do_wake().

- the waker now alters ->activity, but it was too late.
"

So we need get a spinlock to protect this.  And rwsem_is_locked() should
not block, thus we use spin_trylock_irqsave().

[akpm@linux-foundation.org: simplify code]
Reported-by: Brian Behlendorf 
Cc: Ben Woodard 
Cc: David Howells 
Signed-off-by: WANG Cong 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

rwsem-spinlock: remove useless function exports

2009-12-15T16:53:26+00:00

These functions need not to be exported, since no drivers should use them.

__init_rwsem() is an exception, because init_rwsem(), which is a macro,
is used.

Signed-off-by: WANG Cong 
Cc: David Howells 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

lib: remove fastcall from lib/*

2008-02-08T17:22:31+00:00

[akpm@linux-foundation.org: coding-style fixes]
Signed-off-by: Harvey Harrison 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds