linux-stable.git/include/linux/hardirq.h, branch v5.13.2

softirq: Add RT specific softirq accounting

2021-03-17T15:34:08+00:00

RT requires the softirq processing and local bottomhalf disabled regions to
be preemptible. Using the normal preempt count based serialization is
therefore not possible because this implicitely disables preemption.

RT kernels use a per CPU local lock to serialize bottomhalfs. As
local_bh_disable() can nest the lock can only be acquired on the outermost
invocation of local_bh_disable() and released when the nest count becomes
zero. Tasks which hold the local lock can be preempted so its required to
keep track of the nest count per task.

Add a RT only counter to task struct and adjust the relevant macros in
preempt.h.

Signed-off-by: Thomas Gleixner 
Tested-by: Sebastian Andrzej Siewior 
Tested-by: Paul E. McKenney 
Reviewed-by: Frederic Weisbecker 
Acked-by: Peter Zijlstra (Intel) 
Link: https://lore.kernel.org/r/20210309085726.983627589@linutronix.de

irqtime: Move irqtime entry accounting after irq offset incrementation

2020-12-02T19:20:05+00:00

IRQ time entry is currently accounted before HARDIRQ_OFFSET or
SOFTIRQ_OFFSET are incremented. This is convenient to decide to which
index the cputime to account is dispatched.

Unfortunately it prevents tick_irq_enter() from being called under
HARDIRQ_OFFSET because tick_irq_enter() has to be called before the IRQ
entry accounting due to the necessary clock catch up. As a result we
don't benefit from appropriate lockdep coverage on tick_irq_enter().

To prepare for fixing this, move the IRQ entry cputime accounting after
the preempt offset is incremented. This requires the cputime dispatch
code to handle the extra offset.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Thomas Gleixner 
Acked-by: Peter Zijlstra (Intel) 
Link: https://lore.kernel.org/r/20201202115732.27827-5-frederic@kernel.org

x86/entry: Fix NMI vs IRQ state tracking

2020-07-10T10:00:01+00:00

While the nmi_enter() users did
trace_hardirqs_{off_prepare,on_finish}() there was no matching
lockdep_hardirqs_*() calls to complete the picture.

Introduce idtentry_{enter,exit}_nmi() to enable proper IRQ state
tracking across the NMIs.

Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Ingo Molnar 
Link: https://lkml.kernel.org/r/20200623083721.216740948@infradead.org

genirq: Provide __irq_enter/exit_raw()

2020-06-11T13:15:06+00:00

Like __irq_enter/exit() but without time accounting. To be used for "empty"
system vectors like the scheduler IPI to avoid the overhead.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar 
Reviewed-by: Andy Lutomirski 
Link: https://lore.kernel.org/r/20200521202117.671682341@linutronix.de

genirq: Provide irq_enter/exit_rcu()

2020-06-11T13:15:06+00:00

irq_enter()/exit() currently include RCU handling. To properly separate the RCU
handling code, provide variants which contain only the non-RCU related
functionality.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar 
Reviewed-by: Andy Lutomirski 
Link: https://lore.kernel.org/r/20200521202117.567023613@linutronix.de

nmi, tracing: Make hardware latency tracing noinstr safe

2020-06-11T13:15:04+00:00

The hardware latency tracer calls into instrumentable functions. Move the
calls into the RCU watching sections and annotate them.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar 
Acked-by: Andy Lutomirski 
Cc: Paul E. McKenney 
Link: https://lore.kernel.org/r/20200521202116.904176298@linutronix.de
Signed-off-by: Ingo Molnar

rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

2020-05-26T17:04:18+00:00

There will likely be exception handlers that can sleep, which rules
out the usual approach of invoking rcu_nmi_enter() on entry and also
rcu_nmi_exit() on all exit paths.  However, the alternative approach of
just not calling anything can prevent RCU from coaxing quiescent states
from nohz_full CPUs that are looping in the kernel:  RCU must instead
IPI them explicitly.  It would be better to enable the scheduler tick
on such CPUs to interact with RCU in a lighter-weight manner, and this
enabling is one of the things that rcu_nmi_enter() currently does.

What is needed is something that helps RCU coax quiescent states while
not preventing subsequent sleeps.  This commit therefore splits out the
nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter()
logic into a new function named rcu_irq_enter_check_tick().

[ tglx: Renamed the function and made it a nop when context tracking is off ]
[ mingo: Fixed a CONFIG_NO_HZ_FULL assumption, harmonized and fixed all the
         comment blocks and cleaned up rcu_nmi_enter()/exit() definitions. ]

Suggested-by: Andy Lutomirski 
Signed-off-by: Paul E. McKenney 
Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar 
Link: https://lore.kernel.org/r/20200521202116.996113173@linutronix.de

sched,rcu,tracing: Avoid tracing before in_nmi() is correct

2020-05-19T13:51:18+00:00

If a tracer is invoked before in_nmi() becomes true, the tracer can no
longer detect it is called from NMI context and behave correctly.

Therefore change nmi_{enter,exit}() to use __preempt_count_{add,sub}()
as the normal preempt_count_{add,sub}() have a (desired) function
trace entry.

This fixes a potential issue with the current code; when the function-tracer
has stack-tracing enabled __trace_stack() will malfunction when it hits the
preempt_count_add() function entry from NMI context.

Suggested-by: Steven Rostedt (VMware) 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner 
Reviewed-by: Steven Rostedt (VMware) 
Reviewed-by: Alexandre Chartre 
Link: https://lkml.kernel.org/r/20200505134101.434193525@linutronix.de

hardirq/nmi: Allow nested nmi_enter()

2020-05-19T13:51:17+00:00

Since there are already a number of sites (ARM64, PowerPC) that effectively
nest nmi_enter(), make the primitive support this before adding even more.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner 
Acked-by: Marc Zyngier 
Acked-by: Will Deacon 
Cc: Michael Ellerman 
Link: https://lkml.kernel.org/r/20200505134100.864179229@linutronix.de

lockdep: Rename trace_hardirq_{enter,exit}()

2020-03-21T15:03:53+00:00

Continue what commit:

  d820ac4c2fa8 ("locking: rename trace_softirq_[enter|exit] => lockdep_softirq_[enter|exit]")

started, rename these to avoid confusing them with tracepoints.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Thomas Gleixner 
Acked-by: Will Deacon 
Link: https://lkml.kernel.org/r/20200320115859.060481361@infradead.org