linux-stable.git/kernel/irq_work.c, branch v3.17.2

irq_work: Force raised irq work to run on irq work interrupt

2014-10-15T10:29:23+00:00

commit 76a33061b9323b7fdb220ae5fa116c10833ec22e upstream.

The nohz full kick, which restarts the tick when any resource depend
on it, can't be executed anywhere given the operation it does on timers.
If it is called from the scheduler or timers code, chances are that
we run into a deadlock.

This is why we run the nohz full kick from an irq work. That way we make
sure that the kick runs on a virgin context.

However if that's the case when irq work runs in its own dedicated
self-ipi, things are different for the big bunch of archs that don't
support the self triggered way. In order to support them, irq works are
also handled by the timer interrupt as fallback.

Now when irq works run on the timer interrupt, the context isn't blank.
More precisely, they can run in the context of the hrtimer that runs the
tick. But the nohz kick cancels and restarts this hrtimer and cancelling
an hrtimer from itself isn't allowed. This is why we run in an endless
loop:

	Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
	CPU: 2 PID: 7538 Comm: kworker/u8:8 Not tainted 3.16.0+ #34
	Workqueue: btrfs-endio-write normal_work_helper [btrfs]
	 ffff880244c06c88 000000001b486fe1 ffff880244c06bf0 ffffffff8a7f1e37
	 ffffffff8ac52a18 ffff880244c06c78 ffffffff8a7ef928 0000000000000010
	 ffff880244c06c88 ffff880244c06c20 000000001b486fe1 0000000000000000
	Call Trace:
	 ] dump_stack+0x4e/0x7a
	 [] panic+0xd4/0x207
	 [] watchdog_overflow_callback+0x118/0x120
	 [] __perf_event_overflow+0xae/0x350
	 [] ? perf_event_task_disable+0xa0/0xa0
	 [] ? x86_perf_event_set_period+0xbf/0x150
	 [] perf_event_overflow+0x14/0x20
	 [] intel_pmu_handle_irq+0x206/0x410
	 [] perf_event_nmi_handler+0x2b/0x50
	 [] nmi_handle+0xd2/0x390
	 [] ? nmi_handle+0x5/0x390
	 [] ? match_held_lock+0x8/0x1b0
	 [] default_do_nmi+0x72/0x1c0
	 [] do_nmi+0xb8/0x100
	 [] end_repeat_nmi+0x1e/0x2e
	 [] ? match_held_lock+0x8/0x1b0
	 [] ? match_held_lock+0x8/0x1b0
	 [] ? match_held_lock+0x8/0x1b0
	 <] lock_acquired+0xaf/0x450
	 [] ? lock_hrtimer_base.isra.20+0x25/0x50
	 [] _raw_spin_lock_irqsave+0x78/0x90
	 [] ? lock_hrtimer_base.isra.20+0x25/0x50
	 [] lock_hrtimer_base.isra.20+0x25/0x50
	 [] hrtimer_try_to_cancel+0x33/0x1e0
	 [] hrtimer_cancel+0x1a/0x30
	 [] tick_nohz_restart+0x17/0x90
	 [] __tick_nohz_full_check+0xc3/0x100
	 [] nohz_full_kick_work_func+0xe/0x10
	 [] irq_work_run_list+0x44/0x70
	 [] irq_work_run+0x2a/0x50
	 [] update_process_times+0x5b/0x70
	 [] tick_sched_handle.isra.21+0x25/0x60
	 [] tick_sched_timer+0x41/0x60
	 [] __run_hrtimer+0x72/0x470
	 [] ? tick_sched_do_timer+0xb0/0xb0
	 [] hrtimer_interrupt+0x117/0x270
	 [] local_apic_timer_interrupt+0x37/0x60
	 [] smp_apic_timer_interrupt+0x3f/0x50
	 [] apic_timer_interrupt+0x6f/0x80

To fix this we force non-lazy irq works to run on irq work self-IPIs
when available. That ability of the arch to trigger irq work self IPIs
is available with arch_irq_work_has_interrupt().

Reported-by: Catalin Iacob 
Reported-by: Dave Jones 
Acked-by: Peter Zijlstra (Intel) 
Cc: Ingo Molnar 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Greg Kroah-Hartman

irq_work: Remove BUG_ON in irq_work_run()

2014-07-05T09:17:26+00:00

Because of a collision with 8d056c48e486 ("CPU hotplug, smp: flush any
pending IPI callbacks before CPU offline"), which ends up calling
hotplug_cfd()->flush_smp_call_function_queue()->irq_work_run(), which
is not from IRQ context.

And since that already calls irq_work_run() from the hotplug path,
remove our entire hotplug handling.

Reported-by: Stephen Warren 
Tested-by: Stephen Warren 
Reviewed-by: Srivatsa S. Bhat 
Cc: Frederic Weisbecker 
Cc: Linus Torvalds 
Signed-off-by: Peter Zijlstra 
Link: http://lkml.kernel.org/n/tip-busatzs2gvz4v62258agipuf@git.kernel.org
Signed-off-by: Ingo Molnar

irq_work: Implement remote queueing

2014-06-16T14:26:54+00:00

irq work currently only supports local callbacks. However its code
is mostly ready to run remote callbacks and we have some potential user.

The full nohz subsystem currently open codes its own remote irq work
on top of the scheduler ipi when it wants a CPU to reevaluate its next
tick. However this ad hoc solution bloats the scheduler IPI.

Lets just extend the irq work subsystem to support remote queuing on top
of the generic SMP IPI to handle this kind of user. This shouldn't add
noticeable overhead.

Suggested-by: Peter Zijlstra 
Acked-by: Peter Zijlstra 
Cc: Andrew Morton 
Cc: Eric Dumazet 
Cc: Ingo Molnar 
Cc: Kevin Hilman 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

irq_work: Split raised and lazy lists

2014-06-16T14:26:53+00:00

An irq work can be handled from two places: from the tick if the work
carries the "lazy" flag and the tick is periodic, or from a self IPI.

We merge all these works in a single list and we use some per cpu latch
to avoid raising a self-IPI when one is already pending.

Now we could do away with this ugly latch if only the list was only made of
non-lazy works. Just enqueueing a work on the empty list would be enough
to know if we need to raise an IPI or not.

Also we are going to implement remote irq work queuing. Then the per CPU
latch will need to become atomic in the global scope. That's too bad
because, here as well, just enqueueing a work on an empty list of
non-lazy works would be enough to know if we need to raise an IPI or not.

So lets take a way out of this: split the works in two distinct lists,
one for the works that can be handled by the next tick and another
one for those handled by the IPI. Just checking if the latter is empty
when we queue a new work is enough to know if we need to raise an IPI.

Suggested-by: Peter Zijlstra 
Acked-by: Peter Zijlstra 
Cc: Andrew Morton 
Cc: Ingo Molnar 
Cc: Kevin Hilman 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

perf/x86: Warn to early_printk() in case irq_work is too slow

2014-02-21T20:49:07+00:00

On Mon, Feb 10, 2014 at 08:45:16AM -0800, Dave Hansen wrote:
> The reason I coded this up was that NMIs were firing off so fast that
> nothing else was getting a chance to run.  With this patch, at least the
> printk() would come out and I'd have some idea what was going on.

It will start spewing to early_printk() (which is a lot nicer to use
from NMI context too) when it fails to queue the IRQ-work because its
already enqueued.

It does have the false-positive for when two CPUs trigger the warn
concurrently, but that should be rare and some extra clutter on the
early printk shouldn't be a problem.

Cc: hpa@zytor.com
Cc: tglx@linutronix.de
Cc: dzickus@redhat.com
Cc: Dave Hansen 
Cc: mingo@kernel.org
Fixes: 6a02ad66b2c4 ("perf/x86: Push the duration-logging printk() to IRQ context")
Signed-off-by: Peter Zijlstra 
Link: http://lkml.kernel.org/r/20140211150116.GO27965@twins.programming.kicks-ass.net
Signed-off-by: Thomas Gleixner

Merge branch 'nohz/printk-v8' into irq/core

2013-02-04T23:48:46+00:00

Conflicts:
	kernel/irq_work.c

Add support for printk in full dynticks CPU.

* Don't stop tick with irq works pending. This
fix is generally useful and concerns archs that
can't raise self IPIs.

* Flush irq works before CPU offlining.

* Introduce "lazy" irq works that can wait for the
next tick to be executed, unless it's stopped.

* Implement klogd wake up using irq work. This
removes the ad-hoc printk_tick()/printk_needs_cpu()
hooks and make it working even in dynticks mode.

Signed-off-by: Frederic Weisbecker

irq_work: Remove return value from the irq_work_queue() function

2013-02-04T10:50:59+00:00

As no one is using the return value of irq_work_queue(),
so it is better to just make it void.

Signed-off-by: anish kumar 
Acked-by: Steven Rostedt 
[ Fix stale comments, remove now unnecessary __irq_work_queue() intermediate function ]
Signed-off-by: Frederic Weisbecker 
Link: http://lkml.kernel.org/r/1359925703-24304-1-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar

irq_work: Make self-IPIs optable

2012-11-18T00:01:22+00:00

On irq work initialization, let the user choose to define it
as "lazy" or not. "Lazy" means that we don't want to send
an IPI (provided the arch can anyway) when we enqueue this
work but we rather prefer to wait for the next timer tick
to execute our work if possible.

This is going to be a benefit for non-urgent enqueuers
(like printk in the future) that may prefer not to raise
an IPI storm in case of frequent enqueuing on short periods
of time.

Signed-off-by: Frederic Weisbecker 
Acked-by: Steven Rostedt 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Andrew Morton 
Cc: Paul Gortmaker

irq_work: Warn if there's still work on cpu_down

2012-11-17T18:31:16+00:00

If we are in nohz and there's still irq_work to be done when the idle
task is about to go offline, give a nasty warning. Everything should
have been flushed from the CPU_DYING notifier already. Further attempts
to enqueue an irq_work are buggy because irqs are disabled by
__cpu_disable(). The best we can do is to report the issue to the user.

Signed-off-by: Steven Rostedt 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Andrew Morton 
Cc: Paul Gortmaker 
Signed-off-by: Frederic Weisbecker

irq_work: Flush work on CPU_DYING

2012-11-17T18:31:03+00:00

In order not to offline a CPU with pending irq works, flush the
queue from CPU_DYING. The notifier is called by stop_machine on
the CPU that is going down. The code will not be called from irq context
(so things like get_irq_regs() wont work) but I'm not sure what the
requirements are for irq_work in that regard (Peter?). But irqs are
disabled and the CPU is about to go offline. Might as well flush the work.

Signed-off-by: Steven Rostedt 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Andrew Morton 
Cc: Paul Gortmaker 
Signed-off-by: Frederic Weisbecker