linux.git/arch/mips/include/asm/atomic.h, branch v5.6

MIPS: atomic: Deduplicate 32b & 64b read, set, xchg, cmpxchg

2019-10-07T16:42:36+00:00

Remove the remaining duplication between 32b & 64b in asm/atomic.h by
making use of an ATOMIC_OPS() macro to generate:

  - atomic_read()/atomic64_read()
  - atomic_set()/atomic64_set()
  - atomic_cmpxchg()/atomic64_cmpxchg()
  - atomic_xchg()/atomic64_xchg()

This is consistent with the way all other functions in asm/atomic.h are
generated, and ensures consistency between the 32b & 64b functions.

Of note is that this results in the above now being static inline
functions rather than macros.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Unify 32b & 64b sub_if_positive

2019-10-07T16:42:34+00:00

Unify the definitions of atomic_sub_if_positive() &
atomic64_sub_if_positive() using a macro like we do for most other
atomic functions. This allows us to share the implementation ensuring
consistency between the two. Notably this provides the appropriate
loongson3_war barriers in the atomic64_sub_if_positive() case which were
previously missing.

The code is rearranged a little to handle the !kernel_uses_llsc case
first in order to de-indent the LL/SC case & allow us not to go over 80
characters per line.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Use _atomic barriers in atomic_sub_if_positive()

2019-10-07T16:42:32+00:00

Use smp_mb__before_atomic() & smp_mb__after_atomic() in
atomic_sub_if_positive() rather than the equivalent
smp_mb__before_llsc() & smp_llsc_mb(). The former are more standard &
this preps us for avoiding redundant duplicate barriers on Loongson3 in
a later patch.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Emit Loongson3 sync workarounds within asm

2019-10-07T16:42:31+00:00

Generate the sync instructions required to workaround Loongson3 LL/SC
errata within inline asm blocks, which feels a little safer than doing
it from C where strictly speaking the compiler would be well within its
rights to insert a memory access between the separate asm statements we
previously had, containing sync & ll instructions respectively.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Use one macro to generate 32b & 64b functions

2019-10-07T16:42:29+00:00

Cut down on duplication by generalizing the ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros to work for both 32b &
64b atomics, and removing the ATOMIC64_ variants. This ensures
consistency between our atomic_* & atomic64_* functions.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Handle !kernel_uses_llsc first

2019-10-07T16:42:27+00:00

Handle the !kernel_uses_llsc path first in our ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros & return from within the
block. This allows us to de-indent the kernel_uses_llsc path by one
level which will be useful when making further changes.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: atomic: Fix whitespace in ATOMIC_OP macros

2019-10-07T16:42:26+00:00

We define macros in asm/atomic.h which end each line with space
characters before a backslash to continue on the next line. Remove the
space characters leaving tabs as the whitespace used for conformity with
coding convention.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: Unify sc beqz definition

2019-10-07T16:42:13+00:00

We currently duplicate the definition of __scbeqz in asm/atomic.h &
asm/cmpxchg.h. Move it to asm/llsc.h & rename it to __SC_BEQZ to fit
better with the existing __SC macro provided there.

We include a tab in the string in order to avoid the need for users to
indent code any further to include whitespace of their own after the
instruction mnemonic.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

mips/atomic: Fix smp_mb__{before,after}_atomic()

2019-08-31T10:06:02+00:00

Recent probing at the Linux Kernel Memory Model uncovered a
'surprise'. Strongly ordered architectures where the atomic RmW
primitive implies full memory ordering and
smp_mb__{before,after}_atomic() are a simple barrier() (such as MIPS
without WEAK_REORDERING_BEYOND_LLSC) fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it
(surprisingly) does not provide a compiler barrier. This then allows
the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended. Therefore strengthen the atomic
RmW ops to include a compiler barrier.

Reported-by: Andrea Parri 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Paul Burton

mips/atomic: Fix loongson_llsc_mb() wreckage

2019-08-31T10:05:17+00:00

The comment describing the loongson_llsc_mb() reorder case doesn't
make any sense what so ever. Instruction re-ordering is not an SMP
artifact, but rather a CPU local phenomenon. Clarify the comment by
explaining that these issue cause a coherence fail.

For the branch speculation case; if futex_atomic_cmpxchg_inatomic()
needs one at the bne branch target, then surely the normal
__cmpxch_asm() implementation does too. We cannot rely on the
barriers from cmpxchg() because cmpxchg_local() is implemented with
the same macro, and branch prediction and speculation are, too, CPU
local.

Fixes: e02e07e3127d ("MIPS: Loongson: Introduce and use loongson_llsc_mb()")
Cc: Huacai Chen 
Cc: Huang Pei 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Paul Burton