linux.git/arch/powerpc/kernel/eeh_driver.c, branch v4.17

powerpc/eeh: Fix race with driver un/bind

2018-03-31T13:47:45+00:00

The current EEH callbacks can race with a driver unbind. This can
result in a backtraces like this:

  EEH: Frozen PHB#0-PE#1fc detected
  EEH: PE location: S000009, PHB location: N/A
  CPU: 2 PID: 2312 Comm: kworker/u258:3 Not tainted 4.15.6-openpower1 #2
  Workqueue: nvme-wq nvme_reset_work [nvme]
  Call Trace:
    dump_stack+0x9c/0xd0 (unreliable)
    eeh_dev_check_failure+0x420/0x470
    eeh_check_failure+0xa0/0xa4
    nvme_reset_work+0x138/0x1414 [nvme]
    process_one_work+0x1ec/0x328
    worker_thread+0x2e4/0x3a8
    kthread+0x14c/0x154
    ret_from_kernel_thread+0x5c/0xc8
  nvme nvme1: Removing after probe failure status: -19
  
  cpu 0x23: Vector: 300 (Data Access) at [c000000ff50f3800]
      pc: c0080000089a0eb0: nvme_error_detected+0x4c/0x90 [nvme]
      lr: c000000000026564: eeh_report_error+0xe0/0x110
      sp: c000000ff50f3a80
     msr: 9000000000009033
     dar: 400
   dsisr: 40000000
    current = 0xc000000ff507c000
    paca    = 0xc00000000fdc9d80   softe: 0        irq_happened: 0x01
      pid   = 782, comm = eehd
  Linux version 4.15.6-openpower1 (smc@smc-desktop) (gcc version 6.4.0 (Buildroot 2017.11.2-00008-g4b6188e)) #2 SM                                             P Tue Feb 27 12:33:27 PST 2018
  enter ? for help
    eeh_report_error+0xe0/0x110
    eeh_pe_dev_traverse+0xc0/0xdc
    eeh_handle_normal_event+0x184/0x4c4
    eeh_handle_event+0x30/0x288
    eeh_event_handler+0x124/0x170
    kthread+0x14c/0x154
    ret_from_kernel_thread+0x5c/0xc8

The first part is an EEH (on boot), the second half is the resulting
crash. nvme probe starts the nvme_reset_work() worker thread. This
worker thread starts touching the device which see a device error
(EEH) and hence queues up an event in the powerpc EEH worker
thread. nvme_reset_work() then continues and runs
nvme_remove_dead_ctrl_work() which results in unbinding the driver
from the device and hence releases all resources. At the same time,
the EEH worker thread starts doing the EEH .error_detected() driver
callback, which no longer works since the resources have been freed.

This fixes the problem in the same way the generic PCIe AER code (in
drivers/pci/pcie/aer/aerdrv_core.c) does. It makes the EEH code hold
the device_lock() while performing the driver EEH callbacks and
associated code. This ensures either the callbacks are no longer
register, or if they are registered the driver will not be removed
from underneath us.

This has been broken forever. The EEH call backs were first introduced
in 2005 (in 77bd7415610) but it's not clear if a lock was needed back
then.

Fixes: 77bd74156101 ("[PATCH] powerpc: PCI Error Recovery: PPC64 core recovery routines")
Cc: stable@vger.kernel.org # v2.6.16+
Signed-off-by: Michael Neuling 
Reviewed-by: Benjamin Herrenschmidt 
Signed-off-by: Michael Ellerman

powerpc/eeh: Factor out common code eeh_reset_device()

2018-03-27T12:45:14+00:00

The caller will always pass NULL for 'rmv_data' when
'eeh_aware_driver' is true, so the first two calls to
eeh_pe_dev_traverse() can be combined without changing behaviour as
can the two arms of the final 'if' block.

This should not change behaviour.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Remove always-true tests in eeh_reset_device()

2018-03-27T12:45:00+00:00

eeh_reset_device() tests the value of 'bus' more than once but the
only caller, eeh_handle_normal_device() does this test itself and will
never pass NULL.

So, remove the dead tests.

This should not change behaviour.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Clarify arguments to eeh_reset_device()

2018-03-27T12:44:59+00:00

It is currently difficult to understand the behaviour of
eeh_reset_device() due to the way it's parameters are used. In
particular, when 'bus' is NULL, it's value is still necessary so the
same value is looked up again locally under a different name
('frozen_bus') but behaviour is changed.

To clarify this, add a new parameter 'driver_eeh_aware', and have the
caller set it when it would have passed NULL for 'bus' and always pass
a value for 'bus'. Then change any test that was on 'bus' to one on
'!driver_eeh_aware' and replace uses of 'frozen_bus' with 'bus'.

Also update the function's comment.

This should not change behaviour.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Rename frozen_bus to bus in eeh_handle_normal_event()

2018-03-27T12:44:59+00:00

The name "frozen_bus" is misleading: it's not necessarily frozen, it's
just the PE's PCI bus.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Remove misleading test in eeh_handle_normal_event()

2018-03-27T12:44:58+00:00

Remove a test that checks if "frozen_bus" is NULL, because it cannot
have changed since it was tested at the start of the function and so
must be true here.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Manage EEH_PE_RECOVERING inside eeh_handle_normal_event()

2018-03-27T12:44:58+00:00

Currently the EEH_PE_RECOVERING flag for a PE is managed by both the
caller and callee of eeh_handle_normal_event() (among other places not
considered here). This is complicated by the fact that the PE may
or may not have been invalidated by the call.

So move the callee's handling into eeh_handle_normal_event(), which
clarifies it and allows the return type to be changed to void (because
it no longer needs to indicate at the PE has been invalidated).

This should not change behaviour except in eeh_event_handler() where
it was previously possible to cause eeh_pe_state_clear() to be called
on an invalid PE, which is now avoided.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Remove eeh_handle_event()

2018-03-27T12:44:57+00:00

The function eeh_handle_event(pe) does nothing other than switching
between calling eeh_handle_normal_event(pe) and
eeh_handle_special_event(). However it is only called in two places,
one where pe can't be NULL and the other where it must be NULL (see
eeh_event_handler()) so it does nothing but obscure the flow of
control.

So, remove it.

Signed-off-by: Sam Bobroff 
Reviewed-by: Alexey Kardashevskiy 
Signed-off-by: Michael Ellerman

powerpc/eeh: Fix crashes in eeh_report_resume()

2018-02-21T00:12:27+00:00

The notify_resume() callback in eeh_ops is NULL on powernv, leading to
crashes:

  NIP (null)
  LR  eeh_report_resume+0x218/0x220
  Call Trace:
   eeh_report_resume+0x1f0/0x220 (unreliable)
   eeh_pe_dev_traverse+0x98/0x170
   eeh_handle_normal_event+0x3f4/0x650
   eeh_handle_event+0x54/0x380
   eeh_event_handler+0x14c/0x210
   kthread+0x168/0x1b0
   ret_from_kernel_thread+0x5c/0xb4

Fix it by adding a check before calling it.

Fixes: 856e1eb9bdd4 ("PCI/AER: Add uevents in AER and EEH error/resume")
Signed-off-by: Juan J. Alvarez 
Reviewed-by: Bryant G. Ly 
Tested-by: Carol L. Soto 
Reviewed-by: Andrew Donnellan 
Tested-by: Mauro S. M. Rodrigues 
Acked-by: Michael Neuling 
[mpe: Rewrite change log]
Signed-off-by: Michael Ellerman

PCI/AER: Add uevents in AER and EEH error/resume

2018-01-27T09:02:51+00:00

Devices can go offline when erors reported. This patch adds a change
to the kernel object and lets udev know of error. When device resumes,
a change is also set reporting device as online. Therefore, EEH and
AER events are better propagated to user space for PCI devices in all
arches.

Signed-off-by: Bryant G. Ly 
Signed-off-by: Juan J. Alvarez 
Acked-by: Bjorn Helgaas 
Acked-by: Russell Currey 
Signed-off-by: Michael Ellerman