linux.git/arch/powerpc/include/asm/atomic.h, branch v4.12

powerpc: Remove static branch prediction in atomic{, 64}_add_unless

2016-10-04T05:13:13+00:00

I see quite a lot of static branch mispredictions on a simple
web serving workload. The issue is in __atomic_add_unless(), called
from _atomic_dec_and_lock(). There is no obvious common case, so it
is better to let the hardware predict the branch.

Signed-off-by: Anton Blanchard 
Signed-off-by: Michael Ellerman

locking/atomic, arch/powerpc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}{,_relaxed,_acquire,_release}()

2016-06-16T08:48:28+00:00

Implement FETCH-OP atomic primitives, these are very similar to the
existing OP-RETURN primitives we already have, except they return the
value of the atomic variable _before_ modification.

This is especially useful for irreversible operations -- such as
bitops (because it becomes impossible to reconstruct the state prior
to modification).

Tested-by: Boqun Feng 
Signed-off-by: Peter Zijlstra (Intel) 
Cc: Andrew Morton 
Cc: Benjamin Herrenschmidt 
Cc: Linus Torvalds 
Cc: Michael Ellerman 
Cc: Paul E. McKenney 
Cc: Paul Mackerras 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: linux-arch@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Ingo Molnar

powerpc: atomic: Implement acquire/release/relaxed variants for cmpxchg

2016-02-17T13:11:39+00:00

Implement cmpxchg{,64}_relaxed and atomic{,64}_cmpxchg_relaxed, based on
which _release variants can be built.

To avoid superfluous barriers in _acquire variants, we implement these
operations with assembly code rather use __atomic_op_acquire() to build
them automatically.

For the same reason, we keep the assembly implementation of fully
ordered cmpxchg operations.

However, we don't do the similar for _release, because that will require
putting barriers in the middle of ll/sc loops, which is probably a bad
idea.

Note cmpxchg{,64}_relaxed and atomic{,64}_cmpxchg_relaxed are not
compiler barriers.

Signed-off-by: Boqun Feng 
Signed-off-by: Michael Ellerman

powerpc: atomic: Implement acquire/release/relaxed variants for xchg

2016-02-17T13:11:31+00:00

Implement xchg{,64}_relaxed and atomic{,64}_xchg_relaxed, based on these
_relaxed variants, release/acquire variants and fully ordered versions
can be built.

Note that xchg{,64}_relaxed and atomic_{,64}_xchg_relaxed are not
compiler barriers.

Signed-off-by: Boqun Feng 
Signed-off-by: Michael Ellerman

powerpc: atomic: Implement atomic{, 64}__return_ variants

2016-02-17T13:11:21+00:00

On powerpc, acquire and release semantics can be achieved with
lightweight barriers("lwsync" and "ctrl+isync"), which can be used to
implement __atomic_op_{acquire,release}.

For release semantics, since we only need to ensure all memory accesses
that issue before must take effects before the -store- part of the
atomics, "lwsync" is what we only need. On the platform without
"lwsync", "sync" should be used. Therefore in __atomic_op_release() we
use PPC_RELEASE_BARRIER.

For acquire semantics, "lwsync" is what we only need for the similar
reason.  However on the platform without "lwsync", we can use "isync"
rather than "sync" as an acquire barrier. Therefore in
__atomic_op_acquire() we use PPC_ACQUIRE_BARRIER, which is barrier() on
UP, "lwsync" if available and "isync" otherwise.

Implement atomic{,64}_{add,sub,inc,dec}_return_relaxed, and build other
variants with these helpers.

Signed-off-by: Boqun Feng 
Signed-off-by: Michael Ellerman

atomic: Provide atomic_{or,xor,and}

2015-07-27T12:06:24+00:00

Implement atomic logic ops -- atomic_{or,xor,and}.

These will replace the atomic_{set,clear}_mask functions that are
available on some archs.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner

powerpc: Provide atomic_{or,xor,and}

2015-07-27T12:06:23+00:00

Implement atomic logic ops -- atomic_{or,xor,and}.

These will replace the atomic_{set,clear}_mask functions that are
available on some archs.

Acked-by: Benjamin Herrenschmidt 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner

locking,arch,powerpc: Fold atomic_ops

2014-08-14T10:48:11+00:00

Many of the atomic op implementations are the same except for one
instruction; fold the lot into a few CPP macros and reduce LoC.

Requires asm_op because PPC asm is weird :-)

Signed-off-by: Peter Zijlstra 
Cc: Benjamin Herrenschmidt 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Paul Mackerras 
Cc: Mahesh Salgaonkar 
Cc: Paul Gortmaker 
Cc: linuxppc-dev@lists.ozlabs.org
Link: http://lkml.kernel.org/r/20140508135852.713980957@infradead.org
Signed-off-by: Ingo Molnar

arch,powerpc: Convert smp_mb__*()

2014-04-18T12:20:41+00:00

Powerpc allows reordering over its ll/sc implementation. Implement the
two new barriers as appropriate.

Signed-off-by: Peter Zijlstra 
Acked-by: Paul E. McKenney 
Link: http://lkml.kernel.org/n/tip-gg2ffgq32sjgy9b8lj6m3hsc@git.kernel.org
Cc: Benjamin Herrenschmidt 
Cc: Linus Torvalds 
Cc: Mahesh Salgaonkar 
Cc: Paul Gortmaker 
Cc: Paul Mackerras 
Cc: Will Deacon 
Cc: linux-kernel@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Ingo Molnar

atomic: implement generic atomic_dec_if_positive()

2012-10-09T07:22:46+00:00

The x86 implementation of atomic_dec_if_positive is quite generic, so make
it available to all architectures.

This is needed for "swap: add a simple detector for inappropriate swapin
readahead".

[akpm@linux-foundation.org: do the "#define foo foo" trick in the conventional manner]
Signed-off-by: Shaohua Li 
Cc: Stephen Rothwell 
Cc: "David S. Miller" 
Cc: Rik van Riel 
Cc: Ingo Molnar 
Cc: Thomas Gleixner 
Cc: "H. Peter Anvin" 
Cc: Benjamin Herrenschmidt 
Cc: Michal Simek 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds