linux-stable.git/kernel/rcu/tree.h, branch v4.6

Merge commit 'fixes.2015.02.23a' into core/rcu

2016-03-15T08:01:06+00:00

 Conflicts:
	kernel/rcu/tree.c

Signed-off-by: Ingo Molnar

rcu: Use simple wait queues where possible in rcutree

2016-02-25T10:27:16+00:00

As of commit dae6e64d2bcfd ("rcu: Introduce proper blocking to no-CBs kthreads
GP waits") the RCU subsystem started making use of wait queues.

Here we convert all additions of RCU wait queues to use simple wait queues,
since they don't need the extra overhead of the full wait queue features.

Originally this was done for RT kernels[1], since we would get things like...

  BUG: sleeping function called from invalid context at kernel/rtmutex.c:659
  in_atomic(): 1, irqs_disabled(): 1, pid: 8, name: rcu_preempt
  Pid: 8, comm: rcu_preempt Not tainted
  Call Trace:
   [] __might_sleep+0xd0/0xf0
   [] rt_spin_lock+0x24/0x50
   [] __wake_up+0x36/0x70
   [] rcu_gp_kthread+0x4d2/0x680
   [] ? __init_waitqueue_head+0x50/0x50
   [] ? rcu_gp_fqs+0x80/0x80
   [] kthread+0xdb/0xe0
   [] ? finish_task_switch+0x52/0x100
   [] kernel_thread_helper+0x4/0x10
   [] ? __init_kthread_worker+0x60/0x60
   [] ? gs_change+0xb/0xb

...and hence simple wait queues were deployed on RT out of necessity
(as simple wait uses a raw lock), but mainline might as well take
advantage of the more streamline support as well.

[1] This is a carry forward of work from v3.10-rt; the original conversion
was by Thomas on an earlier -rt version, and Sebastian extended it to
additional post-3.10 added RCU waiters; here I've added a commit log and
unified the RCU changes into one, and uprev'd it to match mainline RCU.

Signed-off-by: Daniel Wagner 
Acked-by: Peter Zijlstra (Intel) 
Cc: linux-rt-users@vger.kernel.org
Cc: Boqun Feng 
Cc: Marcelo Tosatti 
Cc: Steven Rostedt 
Cc: Paul Gortmaker 
Cc: Paolo Bonzini 
Cc: "Paul E. McKenney" 
Link: http://lkml.kernel.org/r/1455871601-27484-6-git-send-email-wagi@monom.org
Signed-off-by: Thomas Gleixner

rcu: Do not call rcu_nocb_gp_cleanup() while holding rnp->lock

2016-02-25T10:27:16+00:00

rcu_nocb_gp_cleanup() is called while holding rnp->lock. Currently,
this is okay because the wake_up_all() in rcu_nocb_gp_cleanup() will
not enable the IRQs. lockdep is happy.

By switching over using swait this is not true anymore. swake_up_all()
enables the IRQs while processing the waiters. __do_softirq() can now
run and will eventually call rcu_process_callbacks() which wants to
grap nrp->lock.

Let's move the rcu_nocb_gp_cleanup() call outside the lock before we
switch over to swait.

If we would hold the rnp->lock and use swait, lockdep reports
following:

 =================================
 [ INFO: inconsistent lock state ]
 4.2.0-rc5-00025-g9a73ba0 #136 Not tainted
 ---------------------------------
 inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
 rcu_preempt/8 [HC0[0]:SC0[0]:HE1:SE1] takes:
  (rcu_node_1){+.?...}, at: [] rcu_gp_kthread+0xb97/0xeb0
 {IN-SOFTIRQ-W} state was registered at:
   [] __lock_acquire+0xd5f/0x21e0
   [] lock_acquire+0xdf/0x2b0
   [] _raw_spin_lock_irqsave+0x59/0xa0
   [] rcu_process_callbacks+0x141/0x3c0
   [] __do_softirq+0x14d/0x670
   [] irq_exit+0x104/0x110
   [] smp_apic_timer_interrupt+0x46/0x60
   [] apic_timer_interrupt+0x70/0x80
   [] rq_attach_root+0xa6/0x100
   [] cpu_attach_domain+0x16d/0x650
   [] build_sched_domains+0x942/0xb00
   [] sched_init_smp+0x509/0x5c1
   [] kernel_init_freeable+0x172/0x28f
   [] kernel_init+0xe/0xe0
   [] ret_from_fork+0x3f/0x70
 irq event stamp: 76
 hardirqs last  enabled at (75): [] _raw_spin_unlock_irq+0x30/0x60
 hardirqs last disabled at (76): [] _raw_spin_lock_irq+0x1f/0x90
 softirqs last  enabled at (0): [] copy_process.part.26+0x602/0x1cf0
 softirqs last disabled at (0): [<          (null)>]           (null)
 other info that might help us debug this:
  Possible unsafe locking scenario:
        CPU0
        ----
   lock(rcu_node_1);
   
     lock(rcu_node_1);
  *** DEADLOCK ***
 1 lock held by rcu_preempt/8:
  #0:  (rcu_node_1){+.?...}, at: [] rcu_gp_kthread+0xb97/0xeb0
 stack backtrace:
 CPU: 0 PID: 8 Comm: rcu_preempt Not tainted 4.2.0-rc5-00025-g9a73ba0 #136
 Hardware name: Dell Inc. PowerEdge R820/066N7P, BIOS 2.0.20 01/16/2014
  0000000000000000 000000006d7e67d8 ffff881fb081fbd8 ffffffff818379e0
  0000000000000000 ffff881fb0812a00 ffff881fb081fc38 ffffffff8110813b
  0000000000000000 0000000000000001 ffff881f00000001 ffffffff8102fa4f
 Call Trace:
  [] dump_stack+0x4f/0x7b
  [] print_usage_bug+0x1db/0x1e0
  [] ? save_stack_trace+0x2f/0x50
  [] mark_lock+0x66d/0x6e0
  [] ? check_usage_forwards+0x150/0x150
  [] mark_held_locks+0x78/0xa0
  [] ? _raw_spin_unlock_irq+0x30/0x60
  [] trace_hardirqs_on_caller+0x168/0x220
  [] trace_hardirqs_on+0xd/0x10
  [] _raw_spin_unlock_irq+0x30/0x60
  [] swake_up_all+0xb7/0xe0
  [] rcu_gp_kthread+0xab1/0xeb0
  [] ? trace_hardirqs_on_caller+0xff/0x220
  [] ? _raw_spin_unlock_irq+0x41/0x60
  [] ? rcu_barrier+0x20/0x20
  [] kthread+0x104/0x120
  [] ? _raw_spin_unlock_irq+0x30/0x60
  [] ? kthread_create_on_node+0x260/0x260
  [] ret_from_fork+0x3f/0x70
  [] ? kthread_create_on_node+0x260/0x260

Signed-off-by: Daniel Wagner 
Acked-by: Peter Zijlstra (Intel) 
Cc: linux-rt-users@vger.kernel.org
Cc: Boqun Feng 
Cc: Marcelo Tosatti 
Cc: Steven Rostedt 
Cc: Paul Gortmaker 
Cc: Paolo Bonzini 
Cc: "Paul E. McKenney" 
Link: http://lkml.kernel.org/r/1455871601-27484-5-git-send-email-wagi@monom.org
Signed-off-by: Thomas Gleixner

RCU: Privatize rcu_node::lock

2016-02-24T03:59:54+00:00

In patch:

"rcu: Add transitivity to remaining rcu_node ->lock acquisitions"

All locking operations on rcu_node::lock are replaced with the wrappers
because of the need of transitivity, which indicates we should never
write code using LOCK primitives alone(i.e. without a proper barrier
following) on rcu_node::lock outside those wrappers. We could detect
this kind of misuses on rcu_node::lock in the future by adding __private
modifier on rcu_node::lock.

To privatize rcu_node::lock, unlock wrappers are also needed. Replacing
spinlock unlocks with these wrappers not only privatizes rcu_node::lock
but also makes it easier to figure out critical sections of rcu_node.

This patch adds __private modifier to rcu_node::lock and makes every
access to it wrapped by ACCESS_PRIVATE(). Besides, unlock wrappers are
added and raw_spin_unlock(&rnp->lock) and its friends are replaced with
those wrappers.

Signed-off-by: Boqun Feng 
Signed-off-by: Paul E. McKenney

Merge branches 'doc.2015.12.05a', 'exp.2015.12.07a', 'fixes.2015.12.07a', 'list.2015.12.04b' and 'torture.2015.12.05a' into HEAD

2015-12-08T01:02:54+00:00

doc.2015.12.05a:  Documentation updates
exp.2015.12.07a:  Expedited grace-period updates
fixes.2015.12.07a:  Miscellaneous fixes
list.2015.12.04b:  Linked-list updates
torture.2015.12.05a:  Torture-test updates

rcutorture: Print symbolic name for ->gp_state

2015-12-06T01:58:26+00:00

Currently, ->gp_state is printed as an integer, which slows debugging.
This commit therefore prints a symbolic name in addition to the integer.

Signed-off-by: Paul E. McKenney 
[ paulmck: Updated to fix relational operator called out by Dan Carpenter. ]
[ paulmck: More "const", as suggested by Josh Triplett. ]
Reviewed-by: Josh Triplett

rcu: Reduce expedited GP memory contention via per-CPU variables

2015-12-04T20:26:52+00:00

Currently, the piggybacked-work checks carried out by sync_exp_work_done()
atomically increment a small set of variables (the ->expedited_workdone0,
->expedited_workdone1, ->expedited_workdone2, ->expedited_workdone3
fields in the rcu_state structure), which will form a memory-contention
bottleneck given a sufficiently large number of CPUs concurrently invoking
either synchronize_rcu_expedited() or synchronize_sched_expedited().

This commit therefore moves these for fields to the per-CPU rcu_data
structure, eliminating the memory contention.  The show_rcuexp() function
also changes to sum up each field in the rcu_data structures.

Signed-off-by: Paul E. McKenney

rcu: Clarify role of ->expmaskinitnext

2015-12-04T20:26:50+00:00

Analogy with the ->qsmaskinitnext field might lead one to believe that
->expmaskinitnext tracks online CPUs.  This belief is incorrect: Any CPU
that has ever been online will have its bit set in the ->expmaskinitnext
field.  This commit therefore adds a comment to make this clear, at
least to people who read comments.

Signed-off-by: Paul E. McKenney

rcu: Create transitive rnp->lock acquisition functions

2015-11-23T18:37:35+00:00

Providing RCU's memory-ordering guarantees requires that the rcu_node
tree's locking provide transitive memory ordering, which the Linux kernel's
spinlocks currently do not provide unless smp_mb__after_unlock_lock()
is used.  Having a separate smp_mb__after_unlock_lock() after each and
every lock acquisition is error-prone, hard to read, and a bit annoying,
so this commit provides wrapper functions that pull in the
smp_mb__after_unlock_lock() invocations.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Paul E. McKenney

Merge branches 'fixes.2015.10.06a' and 'exp.2015.10.07a' into HEAD

2015-10-07T23:05:21+00:00

exp.2015.10.07a:  Reduce OS jitter of RCU-sched expedited grace periods.
fixes.2015.10.06a:  Miscellaneous fixes.