linux.git/drivers/dma/idxd/init.c, branch v6.7

dmaengine/idxd: Re-enable kernel workqueue under DMA API

2023-08-09T15:44:39+00:00

Kernel workqueues were disabled due to flawed use of kernel VA and SVA
API. Now that we have the support for attaching PASID to the device's
default domain and the ability to reserve global PASIDs from SVA APIs,
we can re-enable the kernel work queues and use them under DMA API.

We also use non-privileged access for in-kernel DMA to be consistent
with the IOMMU settings. Consequently, interrupt for user privilege is
enabled for work completion IRQs.

Link: https://lore.kernel.org/linux-iommu/20210511194726.GP1002214@nvidia.com/
Tested-by: Tony Zhu 
Reviewed-by: Dave Jiang 
Reviewed-by: Fenghua Yu 
Reviewed-by: Lu Baolu 
Reviewed-by: Jason Gunthorpe 
Acked-by: Vinod Koul 
Signed-off-by: Jacob Pan 
Link: https://lore.kernel.org/r/20230802212427.1497170-9-jacob.jun.pan@linux.intel.com
Signed-off-by: Lu Baolu 
Signed-off-by: Joerg Roedel

Merge tag 'dmaengine-6.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine

2023-05-03T18:11:56+00:00

Pull dmaengine updates from Vinod Koul:
 "New support:

   - Apple admac t8112 device support

   - StarFive JH7110 DMA controller

  Updates:

   - Big pile of idxd updates to support IAA 2.0 device capabilities,
     DSA 2.0 Event Log and completion record faulting features and
     new DSA operations

   - at_xdmac supend & resume updates and driver code cleanup

   - k3-udma supend & resume support

   - k3-psil thread support for J784s4"

* tag 'dmaengine-6.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine: (57 commits)
  dmaengine: idxd: add per wq PRS disable
  dmaengine: idxd: add pid to exported sysfs attribute for opened file
  dmaengine: idxd: expose fault counters to sysfs
  dmaengine: idxd: add a device to represent the file opened
  dmaengine: idxd: add per file user counters for completion record faults
  dmaengine: idxd: process batch descriptor completion record faults
  dmaengine: idxd: add descs_completed field for completion record
  dmaengine: idxd: process user page faults for completion record
  dmaengine: idxd: add idxd_copy_cr() to copy user completion record during page fault handling
  dmaengine: idxd: create kmem cache for event log fault items
  dmaengine: idxd: add per DSA wq workqueue for processing cr faults
  dmanegine: idxd: add debugfs for event log dump
  dmaengine: idxd: add interrupt handling for event log
  dmaengine: idxd: setup event log configuration
  dmaengine: idxd: add event log size sysfs attribute
  dmaengine: idxd: make misc interrupt one shot
  dt-bindings: dma: snps,dw-axi-dmac: constrain the items of resets for JH7110 dma
  dt-bindings: dma: Drop unneeded quotes
  dmaengine: at_xdmac: align declaration of ret with the rest of variables
  dmaengine: at_xdmac: add a warning message regarding for unpaused channels
  ...

Merge branches 'iommu/fixes', 'arm/allwinner', 'arm/exynos', 'arm/mediatek', 'arm/omap', 'arm/renesas', 'arm/rockchip', 'arm/smmu', 'ppc/pamu', 'unisoc', 'x86/vt-d', 'x86/amd', 'core' and 'platform-remove_new' into next

2023-04-14T11:45:50+00:00

dmaengine: idxd: Add enable/disable device IOPF feature

2023-04-13T10:05:46+00:00

The iommu subsystem requires IOMMU_DEV_FEAT_IOPF must be enabled before
and disabled after IOMMU_DEV_FEAT_SVA, if device's I/O page faults rely
on the IOMMU. Add explicit IOMMU_DEV_FEAT_IOPF enabling/disabling in this
driver.

At present, missing IOPF enabling/disabling doesn't cause any real issue,
because the IOMMU driver places the IOPF enabling/disabling in the path
of SVA feature handling. But this may change.

Reviewed-by: Dave Jiang 
Reviewed-by: Fenghua Yu 
Reviewed-by: Kevin Tian 
Acked-by: Vinod Koul 
Signed-off-by: Lu Baolu 
Link: https://lore.kernel.org/r/20230324120234.313643-2-baolu.lu@linux.intel.com
Signed-off-by: Joerg Roedel

dmaengine: idxd: process batch descriptor completion record faults

2023-04-12T17:48:45+00:00

Add event log processing for faulting of user batch descriptor completion
record.

When encountering an event log entry for a page fault on a completion
record, the driver is expected to do the following:
1. If the "first error in batch" bit in event log entry error info is
set, discard any previously recorded errors associated with the
"batch identifier".
2. Fix the page fault according to the fault address in the event log. If
successful, write the completion record to the fault address in user space.
3. If an error is encountered while writing the completion record and it is
associated to a descriptor in the batch, the driver associates the error
with the batch identifier of the event log entry and tracks it until the
event log entry for the corresponding batch desc is encountered.

While processing an event log entry for a batch descriptor with error
indicating that one or more descs in the batch had event log entries,
the driver will do the following before writing the batch completion
record:
1. If the status field of the completion record is 0x1, the driver will
change it to error code 0x5 (one or more operations in batch completed
with status not successful) and changes the result field to 1.
2. If the status is error code 0x6 (page fault on batch descriptor list
address), change the result field to 1.
3. If status is any other value, the completion record is not changed.
4. Clear the recorded error in preparation for next batch with same batch
identifier.

The result field is for user software to determine whether to set the
"Batch Error" flag bit in the descriptor for continuation of partial
batch descriptor completion. See DSA spec 2.0 for additional information.

If no error has been recorded for the batch, the batch completion record is
written to user space as is.

Tested-by: Tony Zhu 
Signed-off-by: Dave Jiang 
Co-developed-by: Fenghua Yu 
Signed-off-by: Fenghua Yu 
Link: https://lore.kernel.org/r/20230407203143.2189681-12-fenghua.yu@intel.com
Signed-off-by: Vinod Koul

dmaengine: idxd: process user page faults for completion record

2023-04-12T17:48:45+00:00

DSA supports page fault handling through PRS. However, the DMA engine
that's processing the descriptor is blocked until the PRS response is
received. Other workqueues sharing the engine are also blocked.
Page fault handing by the driver with PRS disabled can be used to
mitigate the stalling.

With PRS disabled while ATS remain enabled, DSA handles page faults on
a completion record by reporting an event in the event log. In this
instance, the descriptor is completed and the event log contains the
completion record address and the contents of the completion record. Add
support to the event log handling code to fault in the completion record
and copy the content of the completion record to user memory.

A bitmap is introduced to keep track of discarded event log entries. When
the user process initiates ->release() of the char device, it no longer is
interested in any remaining event log entries tied to the relevant wq and
PASID. The driver will mark the event log entry index in the bitmap. Upon
encountering the entries during processing, the event log handler will just
clear the bitmap bit and skip the entry rather than attempt to process the
event log entry.

Tested-by: Tony Zhu 
Signed-off-by: Dave Jiang 
Co-developed-by: Fenghua Yu 
Signed-off-by: Fenghua Yu 
Link: https://lore.kernel.org/r/20230407203143.2189681-10-fenghua.yu@intel.com
Signed-off-by: Vinod Koul

dmaengine: idxd: add idxd_copy_cr() to copy user completion record during page fault handling

2023-04-12T17:48:45+00:00

Define idxd_copy_cr() to copy completion record to fault address in
user address that is found by work queue (wq) and PASID.

It will be used to write the user's completion record that the hardware
device is not able to write due to user completion record page fault.

An xarray is added to associate the PASID and mm with the
struct idxd_user_context so mm can be found by PASID and wq.

It is called when handling the completion record fault in a kernel thread
context. Switch to the mm using kthread_use_vm() and copy the
completion record to the mm via copy_to_user(). Once the copy is
completed, switch back to the current mm using kthread_unuse_mm().

Suggested-by: Christoph Hellwig 
Suggested-by: Jason Gunthorpe 
Suggested-by: Tony Luck 
Tested-by: Tony Zhu 
Signed-off-by: Fenghua Yu 
Reviewed-by: Dave Jiang 
Link: https://lore.kernel.org/r/20230407203143.2189681-9-fenghua.yu@intel.com
Signed-off-by: Vinod Koul

dmaengine: idxd: create kmem cache for event log fault items

2023-04-12T17:48:45+00:00

Add a kmem cache per device for allocating event log fault context. The
context allows an event log entry to be copied and passed to a software
workqueue to be processed. Due to each device can have different sized
event log entry depending on device type, it's not possible to have a
global kmem cache.

Tested-by: Tony Zhu 
Signed-off-by: Dave Jiang 
Co-developed-by: Fenghua Yu 
Signed-off-by: Fenghua Yu 
Link: https://lore.kernel.org/r/20230407203143.2189681-8-fenghua.yu@intel.com
Signed-off-by: Vinod Koul

dmanegine: idxd: add debugfs for event log dump

2023-04-12T17:48:45+00:00

Add debugfs entry to dump the content of the event log for debugging. The
function will dump all non-zero entries in the event log. It will note
which entries are processed and which entries are still pending processing
at the time of the dump. The entries may not always be in chronological
order due to the log is a circular buffer.

Tested-by: Tony Zhu 
Signed-off-by: Dave Jiang 
Co-developed-by: Fenghua Yu 
Signed-off-by: Fenghua Yu 
Link: https://lore.kernel.org/r/20230407203143.2189681-6-fenghua.yu@intel.com
Signed-off-by: Vinod Koul

dmaengine: idxd: setup event log configuration

2023-04-12T17:48:45+00:00

Add setup of event log feature for supported device. Event log addresses
error reporting that was lacking in gen 1 DSA devices where a second error
event does not get reported when a first event is pending software
handling. The event log allows a circular buffer that the device can push
error events to. It is up to the user to create a large enough event log
ring in order to capture the expected events. The evl size can be set in
the device sysfs attribute. By default 64 entries are supported as minimal
when event log is enabled.

Tested-by: Tony Zhu 
Signed-off-by: Dave Jiang 
Co-developed-by: Fenghua Yu 
Signed-off-by: Fenghua Yu 
Link: https://lore.kernel.org/r/20230407203143.2189681-4-fenghua.yu@intel.com
Signed-off-by: Vinod Koul