linux-stable.git/kernel/rcu/tree.h, branch v4.7

Merge branches 'doc.2016.04.19a', 'exp.2016.03.31d', 'fixes.2016.03.31d' and 'torture.2016.04.21a' into HEAD

2016-04-21T20:48:20+00:00

doc.2016.04.19a: Documentation updates
exp.2016.03.31d: Expedited grace-period updates
fixes.2016.03.31d: Miscellaneous fixes
torture.2016.004.21a Torture-test updates

rcu: Awaken grace-period kthread if too long since FQS

2016-03-31T20:34:50+00:00

Recent kernels can fail to awaken the grace-period kthread for
quiescent-state forcing.  This commit is a crude hack that does
a wakeup if a scheduling-clock interrupt sees that it has been
too long since force-quiescent-state (FQS) processing.

Signed-off-by: Paul E. McKenney

rcu: Overlap wakeups with next expedited grace period

2016-03-31T20:34:11+00:00

The current expedited grace-period implementation makes subsequent grace
periods wait on wakeups for the prior grace period.  This does not fit
the dictionary definition of "expedited", so this commit allows these two
phases to overlap.  Doing this requires four waitqueues rather than two
because tasks can now be waiting on the previous, current, and next grace
periods.  The fourth waitqueue makes the bit masking work out nicely.

Signed-off-by: Paul E. McKenney

rcu: Enforce expedited-GP fairness via funnel wait queue

2016-03-31T20:34:08+00:00

The current mutex-based funnel-locking approach used by expedited grace
periods is subject to severe unfairness.  The problem arises when a
few tasks, making a path from leaves to root, all wake up before other
tasks do.  A new task can then follow this path all the way to the root,
which needlessly delays tasks whose grace period is done, but who do
not happen to acquire the lock quickly enough.

This commit avoids this problem by maintaining per-rcu_node wait queues,
along with a per-rcu_node counter that tracks the latest grace period
sought by an earlier task to visit this node.  If that grace period
would satisfy the current task, instead of proceeding up the tree,
it waits on the current rcu_node structure using a pair of wait queues
provided for that purpose.  This decouples awakening of old tasks from
the arrival of new tasks.

If the wakeups prove to be a bottleneck, additional kthreads can be
brought to bear for that purpose.

Signed-off-by: Paul E. McKenney

rcu: Shorten expedited_workdone* to exp_workdone*

2016-03-31T20:34:08+00:00

Just a name change to save a few lines and a bit of typing.

Signed-off-by: Paul E. McKenney

rcu: Remove expedited GP funnel-lock bypass

2016-03-31T20:34:07+00:00

Commit #cdacbe1f91264 ("rcu: Add fastpath bypassing funnel locking")
turns out to be a pessimization at high load because it forces a tree
full of tasks to wait for an expedited grace period that they probably
do not need.  This commit therefore removes this optimization.

Signed-off-by: Paul E. McKenney

Merge commit 'fixes.2015.02.23a' into core/rcu

2016-03-15T08:01:06+00:00

 Conflicts:
	kernel/rcu/tree.c

Signed-off-by: Ingo Molnar

rcu: Use simple wait queues where possible in rcutree

2016-02-25T10:27:16+00:00

As of commit dae6e64d2bcfd ("rcu: Introduce proper blocking to no-CBs kthreads
GP waits") the RCU subsystem started making use of wait queues.

Here we convert all additions of RCU wait queues to use simple wait queues,
since they don't need the extra overhead of the full wait queue features.

Originally this was done for RT kernels[1], since we would get things like...

  BUG: sleeping function called from invalid context at kernel/rtmutex.c:659
  in_atomic(): 1, irqs_disabled(): 1, pid: 8, name: rcu_preempt
  Pid: 8, comm: rcu_preempt Not tainted
  Call Trace:
   [] __might_sleep+0xd0/0xf0
   [] rt_spin_lock+0x24/0x50
   [] __wake_up+0x36/0x70
   [] rcu_gp_kthread+0x4d2/0x680
   [] ? __init_waitqueue_head+0x50/0x50
   [] ? rcu_gp_fqs+0x80/0x80
   [] kthread+0xdb/0xe0
   [] ? finish_task_switch+0x52/0x100
   [] kernel_thread_helper+0x4/0x10
   [] ? __init_kthread_worker+0x60/0x60
   [] ? gs_change+0xb/0xb

...and hence simple wait queues were deployed on RT out of necessity
(as simple wait uses a raw lock), but mainline might as well take
advantage of the more streamline support as well.

[1] This is a carry forward of work from v3.10-rt; the original conversion
was by Thomas on an earlier -rt version, and Sebastian extended it to
additional post-3.10 added RCU waiters; here I've added a commit log and
unified the RCU changes into one, and uprev'd it to match mainline RCU.

Signed-off-by: Daniel Wagner 
Acked-by: Peter Zijlstra (Intel) 
Cc: linux-rt-users@vger.kernel.org
Cc: Boqun Feng 
Cc: Marcelo Tosatti 
Cc: Steven Rostedt 
Cc: Paul Gortmaker 
Cc: Paolo Bonzini 
Cc: "Paul E. McKenney" 
Link: http://lkml.kernel.org/r/1455871601-27484-6-git-send-email-wagi@monom.org
Signed-off-by: Thomas Gleixner

rcu: Do not call rcu_nocb_gp_cleanup() while holding rnp->lock

2016-02-25T10:27:16+00:00

rcu_nocb_gp_cleanup() is called while holding rnp->lock. Currently,
this is okay because the wake_up_all() in rcu_nocb_gp_cleanup() will
not enable the IRQs. lockdep is happy.

By switching over using swait this is not true anymore. swake_up_all()
enables the IRQs while processing the waiters. __do_softirq() can now
run and will eventually call rcu_process_callbacks() which wants to
grap nrp->lock.

Let's move the rcu_nocb_gp_cleanup() call outside the lock before we
switch over to swait.

If we would hold the rnp->lock and use swait, lockdep reports
following:

 =================================
 [ INFO: inconsistent lock state ]
 4.2.0-rc5-00025-g9a73ba0 #136 Not tainted
 ---------------------------------
 inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
 rcu_preempt/8 [HC0[0]:SC0[0]:HE1:SE1] takes:
  (rcu_node_1){+.?...}, at: [] rcu_gp_kthread+0xb97/0xeb0
 {IN-SOFTIRQ-W} state was registered at:
   [] __lock_acquire+0xd5f/0x21e0
   [] lock_acquire+0xdf/0x2b0
   [] _raw_spin_lock_irqsave+0x59/0xa0
   [] rcu_process_callbacks+0x141/0x3c0
   [] __do_softirq+0x14d/0x670
   [] irq_exit+0x104/0x110
   [] smp_apic_timer_interrupt+0x46/0x60
   [] apic_timer_interrupt+0x70/0x80
   [] rq_attach_root+0xa6/0x100
   [] cpu_attach_domain+0x16d/0x650
   [] build_sched_domains+0x942/0xb00
   [] sched_init_smp+0x509/0x5c1
   [] kernel_init_freeable+0x172/0x28f
   [] kernel_init+0xe/0xe0
   [] ret_from_fork+0x3f/0x70
 irq event stamp: 76
 hardirqs last  enabled at (75): [] _raw_spin_unlock_irq+0x30/0x60
 hardirqs last disabled at (76): [] _raw_spin_lock_irq+0x1f/0x90
 softirqs last  enabled at (0): [] copy_process.part.26+0x602/0x1cf0
 softirqs last disabled at (0): [<          (null)>]           (null)
 other info that might help us debug this:
  Possible unsafe locking scenario:
        CPU0
        ----
   lock(rcu_node_1);
   
     lock(rcu_node_1);
  *** DEADLOCK ***
 1 lock held by rcu_preempt/8:
  #0:  (rcu_node_1){+.?...}, at: [] rcu_gp_kthread+0xb97/0xeb0
 stack backtrace:
 CPU: 0 PID: 8 Comm: rcu_preempt Not tainted 4.2.0-rc5-00025-g9a73ba0 #136
 Hardware name: Dell Inc. PowerEdge R820/066N7P, BIOS 2.0.20 01/16/2014
  0000000000000000 000000006d7e67d8 ffff881fb081fbd8 ffffffff818379e0
  0000000000000000 ffff881fb0812a00 ffff881fb081fc38 ffffffff8110813b
  0000000000000000 0000000000000001 ffff881f00000001 ffffffff8102fa4f
 Call Trace:
  [] dump_stack+0x4f/0x7b
  [] print_usage_bug+0x1db/0x1e0
  [] ? save_stack_trace+0x2f/0x50
  [] mark_lock+0x66d/0x6e0
  [] ? check_usage_forwards+0x150/0x150
  [] mark_held_locks+0x78/0xa0
  [] ? _raw_spin_unlock_irq+0x30/0x60
  [] trace_hardirqs_on_caller+0x168/0x220
  [] trace_hardirqs_on+0xd/0x10
  [] _raw_spin_unlock_irq+0x30/0x60
  [] swake_up_all+0xb7/0xe0
  [] rcu_gp_kthread+0xab1/0xeb0
  [] ? trace_hardirqs_on_caller+0xff/0x220
  [] ? _raw_spin_unlock_irq+0x41/0x60
  [] ? rcu_barrier+0x20/0x20
  [] kthread+0x104/0x120
  [] ? _raw_spin_unlock_irq+0x30/0x60
  [] ? kthread_create_on_node+0x260/0x260
  [] ret_from_fork+0x3f/0x70
  [] ? kthread_create_on_node+0x260/0x260

Signed-off-by: Daniel Wagner 
Acked-by: Peter Zijlstra (Intel) 
Cc: linux-rt-users@vger.kernel.org
Cc: Boqun Feng 
Cc: Marcelo Tosatti 
Cc: Steven Rostedt 
Cc: Paul Gortmaker 
Cc: Paolo Bonzini 
Cc: "Paul E. McKenney" 
Link: http://lkml.kernel.org/r/1455871601-27484-5-git-send-email-wagi@monom.org
Signed-off-by: Thomas Gleixner

RCU: Privatize rcu_node::lock

2016-02-24T03:59:54+00:00

In patch:

"rcu: Add transitivity to remaining rcu_node ->lock acquisitions"

All locking operations on rcu_node::lock are replaced with the wrappers
because of the need of transitivity, which indicates we should never
write code using LOCK primitives alone(i.e. without a proper barrier
following) on rcu_node::lock outside those wrappers. We could detect
this kind of misuses on rcu_node::lock in the future by adding __private
modifier on rcu_node::lock.

To privatize rcu_node::lock, unlock wrappers are also needed. Replacing
spinlock unlocks with these wrappers not only privatizes rcu_node::lock
but also makes it easier to figure out critical sections of rcu_node.

This patch adds __private modifier to rcu_node::lock and makes every
access to it wrapped by ACCESS_PRIVATE(). Besides, unlock wrappers are
added and raw_spin_unlock(&rnp->lock) and its friends are replaced with
those wrappers.

Signed-off-by: Boqun Feng 
Signed-off-by: Paul E. McKenney