linux-stable.git/kernel/padata.c, branch v3.18.52

padata: avoid race in reordering

2017-04-18T05:55:51+00:00

commit de5540d088fe97ad583cc7d396586437b32149a5 upstream.

Under extremely heavy uses of padata, crashes occur, and with list
debugging turned on, this happens instead:

[87487.298728] WARNING: CPU: 1 PID: 882 at lib/list_debug.c:33
__list_add+0xae/0x130
[87487.301868] list_add corruption. prev->next should be next
(ffffb17abfc043d0), but was ffff8dba70872c80. (prev=ffff8dba70872b00).
[87487.339011]  [] dump_stack+0x68/0xa3
[87487.342198]  [] ? console_unlock+0x281/0x6d0
[87487.345364]  [] __warn+0xff/0x140
[87487.348513]  [] warn_slowpath_fmt+0x4a/0x50
[87487.351659]  [] __list_add+0xae/0x130
[87487.354772]  [] ? _raw_spin_lock+0x64/0x70
[87487.357915]  [] padata_reorder+0x1e6/0x420
[87487.361084]  [] padata_do_serial+0xa5/0x120

padata_reorder calls list_add_tail with the list to which its adding
locked, which seems correct:

spin_lock(&squeue->serial.lock);
list_add_tail(&padata->list, &squeue->serial.list);
spin_unlock(&squeue->serial.lock);

This therefore leaves only place where such inconsistency could occur:
if padata->list is added at the same time on two different threads.
This pdata pointer comes from the function call to
padata_get_next(pd), which has in it the following block:

next_queue = per_cpu_ptr(pd->pqueue, cpu);
padata = NULL;
reorder = &next_queue->reorder;
if (!list_empty(&reorder->list)) {
       padata = list_entry(reorder->list.next,
                           struct padata_priv, list);
       spin_lock(&reorder->lock);
       list_del_init(&padata->list);
       atomic_dec(&pd->reorder_objects);
       spin_unlock(&reorder->lock);

       pd->processed++;

       goto out;
}
out:
return padata;

I strongly suspect that the problem here is that two threads can race
on reorder list. Even though the deletion is locked, call to
list_entry is not locked, which means it's feasible that two threads
pick up the same padata object and subsequently call list_add_tail on
them at the same time. The fix is thus be hoist that lock outside of
that block.

Signed-off-by: Jason A. Donenfeld 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu 
Signed-off-by: Greg Kroah-Hartman

padata: Fix wrong usage of rcu_dereference()

2013-12-05T13:28:42+00:00

A kernel with enabled lockdep complains about the wrong usage of
rcu_dereference() under a rcu_read_lock_bh() protected region.

  ===============================
  [ INFO: suspicious RCU usage. ]
  3.13.0-rc1+ #126 Not tainted
  -------------------------------
  linux/kernel/padata.c:115 suspicious rcu_dereference_check() usage!

  other info that might help us debug this:

  rcu_scheduler_active = 1, debug_locks = 1
  1 lock held by cryptomgr_test/153:
   #0:  (rcu_read_lock_bh){.+....}, at: [] padata_do_parallel+0x5/0x270

Fix that by using rcu_dereference_bh() instead.

Signed-off-by: Mathias Krause 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: make the sequence counter an atomic_t

2013-10-30T04:02:58+00:00

Using a spinlock to atomically increase a counter sounds wrong -- we've
atomic_t for this!

Also move 'seq_nr' to a different cache line than 'lock' to reduce cache
line trashing. This has the nice side effect of decreasing the size of
struct parallel_data from 192 to 128 bytes for a x86-64 build, e.g.
occupying only two instead of three cache lines.

Those changes results in a 5% performance increase on an IPsec test run
using pcrypt.

Btw. the seq_lock spinlock was never explicitly initialized -- one more
reason to get rid of it.

Signed-off-by: Mathias Krause 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata - Register hotcpu notifier after initialization

2013-08-29T04:37:59+00:00

padata_cpu_callback() takes pinst->lock, to avoid taking
an uninitialized lock, register the notifier after it's
initialization.

Signed-off-by: Richard Weinberger 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata - share code between CPU_ONLINE and CPU_DOWN_FAILED, same to CPU_DOWN_PREPARE and CPU_UP_CANCELED

2013-08-29T04:37:59+00:00

Share code between CPU_ONLINE and CPU_DOWN_FAILED, same to
CPU_DOWN_PREPARE and CPU_UP_CANCELED.

It will fix 2 bugs:

  "not check the return value of __padata_remove_cpu() and __padata_add_cpu()".
  "need add 'break' between CPU_UP_CANCELED and CPU_DOWN_FAILED".

Signed-off-by: Chen Gang 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: use __this_cpu_read per-cpu helper

2012-12-06T09:16:23+00:00

For bottom halves off, __this_cpu_read is better.

Signed-off-by: Shan Wei 
Reviewed-by: Christoph Lameter 
Acked-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: Fix cpu hotplug

2012-03-29T11:52:46+00:00

We don't remove the cpu that went offline from our cpumasks
on cpu hotplug. This got lost somewhere along the line, so
restore it. This fixes a hang of the padata instance on cpu
hotplug.

Signed-off-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: Use the online cpumask as the default

2012-03-29T11:52:46+00:00

We use the active cpumask to determine the superset of cpus
to use for parallelization. However, the active cpumask is
for internal usage of the scheduler and therefore not the
appropriate cpumask for these purposes. So use the online
cpumask instead.

Reported-by: Peter Zijlstra 
Signed-off-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: Add a reference to the api documentation

2012-03-29T11:52:45+00:00

Add a reference to the padata api documentation at Documentation/padata.txt

Suggested-by: Peter Zijlstra 
Signed-off-by: Steffen Klassert 
Signed-off-by: Herbert Xu

padata: Fix race on sequence number wrap

2012-03-14T09:25:56+00:00

When padata_do_parallel() is called from multiple cpus for the same
padata instance, we can get object reordering on sequence number wrap
because testing for sequence number wrap and reseting the sequence
number must happen atomically but is implemented with two atomic
operations. This patch fixes this by converting the sequence number
from atomic_t to an unsigned int and protect the access with a
spin_lock. As a side effect, we get rid of the sequence number wrap
handling because the seqence number wraps back to null now without
the need to do anything.

Signed-off-by: Steffen Klassert 
Signed-off-by: Herbert Xu