linux.git/arch/parisc/kernel/cache.c, branch v5.1

parisc: Optimze cache flush algorithms

2018-10-20T19:10:26+00:00

The attached patch implements three optimizations:

1) Loops in flush_user_dcache_range_asm, flush_kernel_dcache_range_asm,
purge_kernel_dcache_range_asm, flush_user_icache_range_asm, and
flush_kernel_icache_range_asm are unrolled to reduce branch overhead.

2) The static branch prediction for cmpb instructions in pacache.S have
been reviewed and the operand order adjusted where necessary.

3) For flush routines in cache.c, we purge rather flush when we have no
context.  The pdc instruction at level 0 is not required to write back
dirty lines to memory. This provides a performance improvement over the
fdc instruction if the feature is implemented.

Version 2 adds alternative patching.

The patch provides an average improvement of about 2%.

Signed-off-by: John David Anglin 
Signed-off-by: Helge Deller

parisc: Add alternative coding infrastructure

2018-10-17T15:22:26+00:00

This patch adds the necessary code to patch a running kernel at runtime
to improve performance.

The current implementation offers a few optimizations variants:

- When running a SMP kernel on a single UP processor, unwanted assembler
  statements like locking functions are overwritten with NOPs. When
  multiple instructions shall be skipped, one branch instruction is used
  instead of multiple nop instructions.

- In the UP case, some pdtlb and pitlb instructions are patched to
  become pdtlb,l and pitlb,l which only flushes the CPU-local tlb
  entries instead of broadcasting the flush to other CPUs in the system
  and thus may improve performance.

- fic and fdc instructions are skipped if no I- or D-caches are
  installed.  This should speed up qemu emulation and cacheless systems.

- If no cache coherence is needed for IO operations, the relevant fdc
  and sync instructions in the sba and ccio drivers are replaced by
  nops.

- On systems which share I- and D-TLBs and thus don't have a seperate
  instruction TLB, the pitlb instruction is replaced by a nop.

Live-patching is done early in the boot process, just after having run
the system inventory. No drivers are running and thus no external
interrupts should arrive. So the hope is that no TLB exceptions will
occur during the patching. If this turns out to be wrong we will
probably need to do the patching in real-mode.

Signed-off-by: Helge Deller

parisc: Reorder TLB flush timing calculation

2018-10-17T06:18:00+00:00

On boot (mostly reboot), my c8000 sometimes crashes after it prints the
TLB flush threshold.  The lockup is hard.  The front LED flashes red and
the box must be unplugged to reset the error.

I noticed that when the crash occurs the TLB flush threshold is about
one quarter what it is on a successful boot.  If I disabled the
calculation, the crash didn't occur.  There also seemed to be a timing
dependency affecting the crash.  I finally realized that the
flush_tlb_all() timing test runs just after the secondary CPUs are
started.  There seems to be a problem with running flush_tlb_all() too
soon after the CPUs are started.

The timing for the range test always seemed okay.  So, I reversed the
order of the two timing tests and I haven't had a crash at this point so
far.

I added a couple of information messages which I have left to help with
diagnosis if the problem should appear on another machine.

This version reduces the minimum TLB flush threshold to 16 KiB.

Signed-off-by: John David Anglin 
Signed-off-by: Helge Deller

parisc: Move cache flush functions into .text.hot section

2018-04-11T09:40:35+00:00

and move the disable_sr_hashing() C and assembly functions into the
.init section.

Signed-off-by: Helge Deller

mm: fix races between swapoff and flush dcache

2018-04-06T04:36:26+00:00

Thanks to commit 4b3ef9daa4fc ("mm/swap: split swap cache into 64MB
trunks"), after swapoff the address_space associated with the swap
device will be freed.  So page_mapping() users which may touch the
address_space need some kind of mechanism to prevent the address_space
from being freed during accessing.

The dcache flushing functions (flush_dcache_page(), etc) in architecture
specific code may access the address_space of swap device for anonymous
pages in swap cache via page_mapping() function.  But in some cases
there are no mechanisms to prevent the swap device from being swapoff,
for example,

  CPU1					CPU2
  __get_user_pages()			swapoff()
    flush_dcache_page()
      mapping = page_mapping()
        ...				  exit_swap_address_space()
        ...				    kvfree(spaces)
        mapping_mapped(mapping)

The address space may be accessed after being freed.

But from cachetlb.txt and Russell King, flush_dcache_page() only care
about file cache pages, for anonymous pages, flush_anon_page() should be
used.  The implementation of flush_dcache_page() in all architectures
follows this too.  They will check whether page_mapping() is NULL and
whether mapping_mapped() is true to determine whether to flush the
dcache immediately.  And they will use interval tree (mapping->i_mmap)
to find all user space mappings.  While mapping_mapped() and
mapping->i_mmap isn't used by anonymous pages in swap cache at all.

So, to fix the race between swapoff and flush dcache, __page_mapping()
is add to return the address_space for file cache pages and NULL
otherwise.  All page_mapping() invoking in flush dcache functions are
replaced with page_mapping_file().

[akpm@linux-foundation.org: simplify page_mapping_file(), per Mike]
Link: http://lkml.kernel.org/r/20180305083634.15174-1-ying.huang@intel.com
Signed-off-by: "Huang, Ying" 
Reviewed-by: Andrew Morton 
Cc: Minchan Kim 
Cc: Michal Hocko 
Cc: Johannes Weiner 
Cc: Mel Gorman 
Cc: Dave Hansen 
Cc: Chen Liqin 
Cc: Russell King 
Cc: Yoshinori Sato 
Cc: "James E.J. Bottomley" 
Cc: Guan Xuetao 
Cc: "David S. Miller" 
Cc: Chris Zankel 
Cc: Vineet Gupta 
Cc: Ley Foon Tan 
Cc: Ralf Baechle 
Cc: Andi Kleen 
Cc: Mike Rapoport 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

parisc: Handle case where flush_cache_range is called with no context

2018-03-17T10:49:39+00:00

Just when I had decided that flush_cache_range() was always called with
a valid context, Helge reported two cases where the
"BUG_ON(!vma->vm_mm->context);" was hit on the phantom buildd:

 kernel BUG at /mnt/sdb6/linux/linux-4.15.4/arch/parisc/kernel/cache.c:587!
 CPU: 1 PID: 3254 Comm: kworker/1:2 Tainted: G D 4.15.0-1-parisc64-smp #1 Debian 4.15.4-1+b1
 Workqueue: events free_ioctx
  IAOQ[0]: flush_cache_range+0x164/0x168
  IAOQ[1]: flush_cache_page+0x0/0x1c8
  RP(r2): unmap_page_range+0xae8/0xb88
 Backtrace:
  [<00000000404a6980>] unmap_page_range+0xae8/0xb88
  [<00000000404a6ae0>] unmap_single_vma+0xc0/0x188
  [<00000000404a6cdc>] zap_page_range_single+0x134/0x1f8
  [<00000000404a702c>] unmap_mapping_range+0x1cc/0x208
  [<0000000040461518>] truncate_pagecache+0x98/0x108
  [<0000000040461624>] truncate_setsize+0x9c/0xb8
  [<00000000405d7f30>] put_aio_ring_file+0x80/0x100
  [<00000000405d803c>] aio_free_ring+0x8c/0x290
  [<00000000405d82c0>] free_ioctx+0x80/0x180
  [<0000000040284e6c>] process_one_work+0x21c/0x668
  [<00000000402854c4>] worker_thread+0x20c/0x778
  [<0000000040291d44>] kthread+0x2d4/0x2e0
  [<0000000040204020>] end_fault_vector+0x20/0xc0

This indicates that we need to handle the no context case in
flush_cache_range() as we do in flush_cache_mm().

In thinking about this, I realized that we don't need to flush the TLB
when there is no context.  So, I added context checks to the large flush
cases in flush_cache_mm() and flush_cache_range().  The large flush case
occurs frequently in flush_cache_mm() and the change should improve fork
performance.

The v2 version of this change removes the BUG_ON from flush_cache_page()
by skipping the TLB flush when there is no context.  I also added code
to flush the TLB in flush_cache_mm() and flush_cache_range() when we
have a context that's not current.  Now all three routines handle TLB
flushes in a similar manner.

Signed-off-by: John David Anglin 
Cc: stable@vger.kernel.org # 4.9+
Signed-off-by: Helge Deller

parisc: Fix ordering of cache and TLB flushes

2018-03-02T09:03:28+00:00

The change to flush_kernel_vmap_range() wasn't sufficient to avoid the
SMP stalls.  The problem is some drivers call these routines with
interrupts disabled.  Interrupts need to be enabled for flush_tlb_all()
and flush_cache_all() to work.  This version adds checks to ensure
interrupts are not disabled before calling routines that need IPI
interrupts.  When interrupts are disabled, we now drop into slower code.

The attached change fixes the ordering of cache and TLB flushes in
several cases.  When we flush the cache using the existing PTE/TLB
entries, we need to flush the TLB after doing the cache flush.  We don't
need to do this when we flush the entire instruction and data caches as
these flushes don't use the existing TLB entries.  The same is true for
tmpalias region flushes.

The flush_kernel_vmap_range() and invalidate_kernel_vmap_range()
routines have been updated.

Secondly, we added a new purge_kernel_dcache_range_asm() routine to
pacache.S and use it in invalidate_kernel_vmap_range().  Nominally,
purges are faster than flushes as the cache lines don't have to be
written back to memory.

Hopefully, this is sufficient to resolve the remaining problems due to
cache speculation.  So far, testing indicates that this is the case.  I
did work up a patch using tmpalias flushes, but there is a performance
hit because we need the physical address for each page, and we also need
to sequence access to the tmpalias flush code.  This increases the
probability of stalls.

Signed-off-by: John David Anglin 
Cc: stable@vger.kernel.org # 4.9+
Signed-off-by: Helge Deller

parisc: Handle vma's whose context is not current in flush_cache_range

2017-07-31T06:22:33+00:00

In testing James' patch to drivers/parisc/pdc_stable.c, I hit the BUG
statement in flush_cache_range() during a system shutdown:

kernel BUG at arch/parisc/kernel/cache.c:595!
CPU: 2 PID: 6532 Comm: kworker/2:0 Not tainted 4.13.0-rc2+ #1
Workqueue: events free_ioctx

 IAOQ[0]: flush_cache_range+0x144/0x148
 IAOQ[1]: flush_cache_page+0x0/0x1a8
 RP(r2): flush_cache_range+0xec/0x148
Backtrace:
 [<00000000402910ac>] unmap_page_range+0x84/0x880
 [<00000000402918f4>] unmap_single_vma+0x4c/0x60
 [<0000000040291a18>] zap_page_range_single+0x110/0x160
 [<0000000040291c34>] unmap_mapping_range+0x174/0x1a8
 [<000000004026ccd8>] truncate_pagecache+0x50/0xa8
 [<000000004026cd84>] truncate_setsize+0x54/0x70
 [<000000004033d534>] put_aio_ring_file+0x44/0xb0
 [<000000004033d5d8>] aio_free_ring+0x38/0x140
 [<000000004033d714>] free_ioctx+0x34/0xa8
 [<00000000401b0028>] process_one_work+0x1b8/0x4d0
 [<00000000401b04f4>] worker_thread+0x1b4/0x648
 [<00000000401b9128>] kthread+0x1b0/0x208
 [<0000000040150020>] end_fault_vector+0x20/0x28
 [<0000000040639518>] nf_ip_reroute+0x50/0xa8
 [<0000000040638ed0>] nf_ip_route+0x10/0x78
 [<0000000040638c90>] xfrm4_mode_tunnel_input+0x180/0x1f8

CPU: 2 PID: 6532 Comm: kworker/2:0 Not tainted 4.13.0-rc2+ #1
Workqueue: events free_ioctx
Backtrace:
 [<0000000040163bf0>] show_stack+0x20/0x38
 [<0000000040688480>] dump_stack+0xa8/0x120
 [<0000000040163dc4>] die_if_kernel+0x19c/0x2b0
 [<0000000040164d0c>] handle_interruption+0xa24/0xa48

This patch modifies flush_cache_range() to handle non current contexts.
In as much as this occurs infrequently, the simplest approach is to
flush the entire cache when this happens.

Signed-off-by: John David Anglin 
Cc: stable@vger.kernel.org # 4.9+
Signed-off-by: Helge Deller

parisc: Extend disabled preemption in copy_user_page

2017-07-25T21:36:54+00:00

It's always bothered me that we only disable preemption in
copy_user_page around the call to flush_dcache_page_asm.
This patch extends this to after the copy.

Signed-off-by: John David Anglin 
Cc: stable@vger.kernel.org # 4.9+
Signed-off-by: Helge Deller

parisc: Prevent TLB speculation on flushed pages on CPUs that only support equivalent aliases

2017-07-25T21:36:53+00:00

Helge noticed that we flush the TLB page in flush_cache_page but not in
flush_cache_range or flush_cache_mm.

For a long time, we have had random segmentation faults building
packages on machines with PA8800/8900 processors.  These machines only
support equivalent aliases.  We don't see these faults on machines that
don't require strict coherency.  So, it appears TLB speculation
sometimes leads to cache corruption on machines that require coherency.

This patch adds TLB flushes to flush_cache_range and flush_cache_mm when
coherency is required.  We only flush the TLB in flush_cache_page when
coherency is required.

The patch also optimizes flush_cache_range.  It turns out we always have
the right context to use flush_user_dcache_range_asm and
flush_user_icache_range_asm.

The patch has been tested for some time on rp3440, rp3410 and A500-44.
It's been boot tested on c8000.  No random segmentation faults were
observed during testing.

Signed-off-by: John David Anglin 
Cc: stable@vger.kernel.org # 4.9+
Signed-off-by: Helge Deller