linux.git/drivers/virtio/virtio_ring.c, branch v6.5

virtio: add VIRTIO_F_NOTIFICATION_DATA feature support

2023-04-21T07:02:35+00:00

According to VirtIO spec v1.2, VIRTIO_F_NOTIFICATION_DATA feature
indicates that the driver passes extra data along with the queue
notifications.

In a split queue case, the extra data is 16-bit available index. In a
packed queue case, the extra data is 1-bit wrap counter and 15-bit
available index.

Add support for this feature for MMIO, channel I/O and modern PCI
transports.

Signed-off-by: Viktor Prutyanov 
Acked-by: Jason Wang 
Reviewed-by: Xuan Zhuo 
Message-Id: <20230413081855.36643-2-alvaro.karsz@solid-run.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: don't update event idx on get_buf

2023-04-21T07:02:34+00:00

In virtio_net, if we disable napi_tx, when we trigger a tx interrupt,
the vq->event_triggered will be set to true. It is then never reset
until we explicitly call virtqueue_enable_cb_delayed or
virtqueue_enable_cb_prepare.

If we disable the napi_tx, virtqueue_enable_cb* will only be called when
the tx ring is getting relatively empty.

Since event_triggered is true, VRING_AVAIL_F_NO_INTERRUPT or
VRING_PACKED_EVENT_FLAG_DISABLE will not be set. As a result we update
vring_used_event(&vq->split.vring) or vq->packed.vring.driver->off_wrap
every time we call virtqueue_get_buf_ctx. This causes more interrupts.

To summarize:
1) event_triggered was set to true in vring_interrupt()
2) after this nothing will happen in virtqueue_disable_cb() so
   VRING_AVAIL_F_NO_INTERRUPT is not set in avail_flags_shadow
3) virtqueue_get_buf_ctx_split() will still think the cb is enabled
   and then it will publish a new event index

To fix:
update VRING_AVAIL_F_NO_INTERRUPT or VRING_PACKED_EVENT_FLAG_DISABLE in
the vq when we call virtqueue_disable_cb even when event_triggered is
true.

Tested with iperf:
iperf3 tcp stream:
vm1 -----------------> vm2
vm2 just receives tcp data stream from vm1, and sends acks to vm1,
there are many tx interrupts in vm2.
with the patch applied there are just a few tx interrupts.

v2->v3:
-update the interrupt disable flag even with the event_triggered is set,
-instead of checking whether event_triggered is set in
-virtqueue_get_buf_ctx_{packed/split}, will cause the drivers  which have
-not called virtqueue_{enable/disable}_cb to miss notifications.

v3->v4:
-remove change for
-"if (vq->packed.event_flags_shadow != VRING_PACKED_EVENT_FLAG_DISABLE)"
-in virtqueue_disable_cb_packed

Fixes: 8d622d21d248 ("virtio: fix up virtio_disable_cb")
Signed-off-by: Albert Huang 
Signed-off-by: Michael S. Tsirkin 
Signed-off-by: Jason Wang 
Message-Id: <20230329102300.61000-1-huangjie.albert@bytedance.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: Use const to annotate read-only pointer params

2023-04-21T07:02:30+00:00

Add const to make the read-only pointer parameters clear, similar to
many existing functions.

To implement this change, the commit also introduces the use of
`container_of_const` to implement `to_vvq`, which ensures the const-ness
of read-only parameters and avoids accidental modification of their
members.

Signed-off-by: Feng Liu 
Reviewed-by: Jiri Pirko 
Reviewed-by: Parav Pandit 
Reviewed-by: Gavin Li 
Reviewed-by: Bodong Wang 

Message-Id: <20230310053428.3376-4-feliu@nvidia.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: Avoid using inline for small functions

2023-04-21T07:02:30+00:00

According to kernel coding style [1], defining inline functions is not
necessary and beneficial for simple functions. Hence clean up the code
by removing the inline keyword.

It is verified with GCC 12.2.0, the generated code with/without inline
is same. Additionally tested with pktgen and iperf, and verified the
result, the pps test results are the same in the cases of with/without
inline.

Iperf and pps of pktgen for virtio-net didn't change before and after
the change.

[1]
https://www.kernel.org/doc/html/v6.2-rc3/process/coding-style.html#the-inline-disease

Signed-off-by: Feng Liu 
Reviewed-by: Jiri Pirko 
Reviewed-by: Parav Pandit 
Reviewed-by: Gavin Li 
Reviewed-by: Bodong Wang 
Reviewed-by: David Edmondson 
Message-Id: <20230310053428.3376-3-feliu@nvidia.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: per virtqueue dma device

2023-02-21T00:26:58+00:00

This patch introduces a per virtqueue dma device. This will be used
for virtio devices whose virtqueue are backed by different underlayer
devices.

One example is the vDPA that where the control virtqueue could be
implemented through software mediation.

Some of the work are actually done before since the helper like
vring_dma_device(). This work left are:

- Let vring_dma_device() return the per virtqueue dma device instead
  of the vdev's parent.
- Allow passing a dma_device when creating the virtqueue through a new
  helper, old vring creation helper will keep using vdev's parent.

Reviewed-by: Eli Cohen 
Tested-by: Eli Cohen 
Signed-off-by: Jason Wang 
Message-Id: <20230119061525.75068-2-jasowang@redhat.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: use helper function is_power_of_2()

2022-12-28T10:28:09+00:00

Use helper function is_power_of_2() to check if num is power of two.
Minor readability improvement.

Signed-off-by: Shaoqin Huang 
Message-Id: <20221021062734.228881-3-shaoqin.huang@intel.com>
Signed-off-by: Michael S. Tsirkin 
Reviewed-by: Xuan Zhuo

Merge tag 'mm-stable-2022-10-08' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2022-10-11T00:53:04+00:00

Pull MM updates from Andrew Morton:

- Yu Zhao's Multi-Gen LRU patches are here. They've been under test in
linux-next for a couple of months without, to my knowledge, any
negative reports (or any positive ones, come to that).

- Also the Maple Tree from Liam Howlett. An overlapping range-based
tree for vmas. It it apparently slightly more efficient in its own
right, but is mainly targeted at enabling work to reduce mmap_lock
contention.

Liam has identified a number of other tree users in the kernel which
could be beneficially onverted to mapletrees.

Yu Zhao has identified a hard-to-hit but "easy to fix" lockdep splat
at [1]. This has yet to be addressed due to Liam's unfortunately
timed vacation. He is now back and we'll get this fixed up.

- Dmitry Vyukov introduces KMSAN: the Kernel Memory Sanitizer. It uses
clang-generated instrumentation to detect used-unintialized bugs down
to the single bit level.

KMSAN keeps finding bugs. New ones, as well as the legacy ones.

- Yang Shi adds a userspace mechanism (madvise) to induce a collapse of
memory into THPs.

- Zach O'Keefe has expanded Yang Shi's madvise(MADV_COLLAPSE) to
support file/shmem-backed pages.

- userfaultfd updates from Axel Rasmussen

- zsmalloc cleanups from Alexey Romanov

- cleanups from Miaohe Lin: vmscan, hugetlb_cgroup, hugetlb and
memory-failure

- Huang Ying adds enhancements to NUMA balancing memory tiering mode's
page promotion, with a new way of detecting hot pages.

- memcg updates from Shakeel Butt: charging optimizations and reduced
memory consumption.

- memcg cleanups from Kairui Song.

- memcg fixes and cleanups from Johannes Weiner.

- Vishal Moola provides more folio conversions

- Zhang Yi removed ll_rw_block() :(

- migration enhancements from Peter Xu

- migration error-path bugfixes from Huang Ying

- Aneesh Kumar added ability for a device driver to alter the memory
tiering promotion paths. For optimizations by PMEM drivers, DRM
drivers, etc.

- vma merging improvements from Jakub Matěn.

- NUMA hinting cleanups from David Hildenbrand.

- xu xin added aditional userspace visibility into KSM merging
activity.

- THP & KSM code consolidation from Qi Zheng.

- more folio work from Matthew Wilcox.

- KASAN updates from Andrey Konovalov.

- DAMON cleanups from Kaixu Xia.

- DAMON work from SeongJae Park: fixes, cleanups.

- hugetlb sysfs cleanups from Muchun Song.

- Mike Kravetz fixes locking issues in hugetlbfs and in hugetlb core.

Link: https://lkml.kernel.org/r/CAOUHufZabH85CeUN-MEMgL8gJGzJEWUrkiM58JkTbBhh-jew0Q@mail.gmail.com [1]

* tag 'mm-stable-2022-10-08' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (555 commits)
hugetlb: allocate vma lock for all sharable vmas
hugetlb: take hugetlb vma_lock when clearing vma_lock->vma pointer
hugetlb: fix vma lock handling during split vma and range unmapping
mglru: mm/vmscan.c: fix imprecise comments
mm/mglru: don't sync disk for each aging cycle
mm: memcontrol: drop dead CONFIG_MEMCG_SWAP config symbol
mm: memcontrol: use do_memsw_account() in a few more places
mm: memcontrol: deprecate swapaccounting=0 mode
mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled
mm/secretmem: remove reduntant return value
mm/hugetlb: add available_huge_pages() func
mm: remove unused inline functions from include/linux/mm_inline.h
selftests/vm: add selftest for MADV_COLLAPSE of uffd-minor memory
selftests/vm: add file/shmem MADV_COLLAPSE selftest for cleared pmd
selftests/vm: add thp collapse shmem testing
selftests/vm: add thp collapse file and tmpfs testing
selftests/vm: modularize thp collapse memory operations
selftests/vm: dedup THP helpers
mm/khugepaged: add tracepoint to hpage_collapse_scan_file()
mm/madvise: add file and shmem support to MADV_COLLAPSE
...

virtio_ring: make vring_alloc_queue_packed prettier

2022-10-07T13:32:40+00:00

Add some spaces to vring_alloc_queue(make it look prettier).

Signed-off-by: Deming Wang 
Message-Id: <20220926183306.4535-1-wangdeming@inspur.com>
Signed-off-by: Michael S. Tsirkin

virtio_ring: split: Operators use unified style

2022-10-07T13:32:40+00:00

The operators of vring_alloc_queue_split should use the unified style.Add
space for the '|' ,make it be looked more pretty.

Signed-off-by: Deming Wang 
Message-Id: <20220926022202.1516-1-wangdeming@inspur.com>
Signed-off-by: Michael S. Tsirkin

virtio: kmsan: check/unpoison scatterlist in vring_map_one_sg()

2022-10-03T21:03:22+00:00

If vring doesn't use the DMA API, KMSAN is unable to tell whether the
memory is initialized by hardware.  Explicitly call kmsan_handle_dma()
from vring_map_one_sg() in this case to prevent false positives.

Link: https://lkml.kernel.org/r/20220915150417.722975-23-glider@google.com
Signed-off-by: Alexander Potapenko 
Acked-by: Michael S. Tsirkin 
Cc: Alexander Viro 
Cc: Alexei Starovoitov 
Cc: Andrey Konovalov 
Cc: Andrey Konovalov 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Christoph Hellwig 
Cc: Christoph Lameter 
Cc: David Rientjes 
Cc: Dmitry Vyukov 
Cc: Eric Biggers 
Cc: Eric Biggers 
Cc: Eric Dumazet 
Cc: Greg Kroah-Hartman 
Cc: Herbert Xu 
Cc: Ilya Leoshkevich 
Cc: Ingo Molnar 
Cc: Jens Axboe 
Cc: Joonsoo Kim 
Cc: Kees Cook 
Cc: Marco Elver 
Cc: Mark Rutland 
Cc: Matthew Wilcox 
Cc: Pekka Enberg 
Cc: Peter Zijlstra 
Cc: Petr Mladek 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Thomas Gleixner 
Cc: Vasily Gorbik 
Cc: Vegard Nossum 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton