linux-stable.git/arch/mips/include/asm/pgtable.h, branch linux-3.19.y

MIPS: HTW: Prevent accidental HTW start due to nested htw_{start, stop}

2015-03-06T22:57:37+00:00

commit ed4cbc81addbc076b016c5b979fd1a02f0897f0a upstream.

activate_mm() and switch_mm() call get_new_mmu_context() which in turn
can enable the HTW before the entryhi is changed with the new ASID.
Since the latter will enable the HTW in local_flush_tlb_all(),
then there is a small timing window where the HTW is running with the
new ASID but with an old pgd since the TLBMISS_HANDLER_SETUP_PGD
hasn't assigned a new one yet. In order to prevent that, we introduce a
simple htw counter to avoid starting HTW accidentally due to nested
htw_{start,stop}() sequences. Moreover, since various IPI calls can
enforce TLB flushing operations on a different core, such an operation
may interrupt another htw_{stop,start} in progress leading inconsistent
updates of the htw_seq variable. In order to avoid that, we disable the
interrupts whenever we update that variable.

Signed-off-by: Markos Chandras 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9118/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

MIPS: asm: pgtable: Prevent HTW race when updating PTEs

2015-03-06T22:57:28+00:00

commit fde3538a8a711aedf1173ecb2d45aed868f51c97 upstream.

Whenever we modify a page table entry, we need to ensure that the HTW
will not fetch a stable entry. And for that to happen we need to ensure
that HTW is stopped before we modify the said entry otherwise the HTW
may already be in the process of reading that entry and fetching the
old information. As a result of which, we replace the htw_reset() calls
with htw_{stop,start} in more appropriate places. This also removes the
remaining users of htw_reset() and as a result we drop that macro

Signed-off-by: Markos Chandras 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9116/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

MIPS: asm: pgtable: Add c0 hazards on HTW start/stop sequences

2015-03-06T22:57:28+00:00

commit 461d1597ffad7a826f8aaa63ab0727c37b632e34 upstream.

When we use htw_{start,stop}() outside of htw_reset(), we need
to ensure that c0 changes have been propagated properly before
we attempt to continue with subsequence memory operations.

Signed-off-by: Markos Chandras 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9114/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

MIPS: Cosmetic cleanups of page table headers.

2014-11-24T21:47:31+00:00

   * Clean up white spaces and tabs.
   * Remove _PAGE_R4KBUG which is no longer used.
   * Get rid of hardcoded values and calculate shifts and
     masks where possible.

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8457/
Signed-off-by: Ralf Baechle

MIPS: Replace use of phys_t with phys_addr_t.

2014-11-24T21:47:31+00:00

Signed-off-by: Ralf Baechle

MIPS: Replace MIPS-specific 64BIT_PHYS_ADDR with generic PHYS_ADDR_T_64BIT

2014-11-24T21:46:44+00:00

Signed-off-by: Ralf Baechle

MIPS: pgtable.h: Implement the pgprot_writecombine function for MIPS

2014-09-22T11:35:53+00:00

Previously, the pgprot_writecombine function was simply defined
as pgprot_uncached in include/asm-generic/pgtable.h. This is not
optimal for cores that can actually do write-combine memory writes
so define this function to take into account the core's cache coherency
attribute to achieve such behavior.

Signed-off-by: Markos Chandras 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7403/
Signed-off-by: Ralf Baechle

MIPS: Remove race window in page fault handling

2014-08-19T11:31:09+00:00

Multicore MIPSes without I/D hardware coherency suffered from a race
condition in the page fault handler. The page table entry was
published before any pending lazy D-cache flush was committed, hence
it allowed execution of stale page cache data by other VPEs in the
system.

To make the cache handling safe we need to perform flushing already in
the set_pte_at function. MIPSes without coherent I-caches can get a
small increase in flushes due to the unavailability of the execute
flag in set_pte_at.

[ralf@linux-mips.org: outlining set_pte_at() saves a good k in a test
build, so I moved its definition from pgtable.h to cache.c.]

Signed-off-by: Lars Persson 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7511/
Signed-off-by: Ralf Baechle

MIPS: mm: Use the Hardware Page Table Walker if the core supports it

2014-08-01T22:06:39+00:00

The Hardware Page Table Walker aims to speed up TLB refill exceptions
by handling them in the hardware level instead of having a software
TLB refill handler. However, a TLB refill exception can still be
thrown in certain cases such as, synchronus exceptions, or address
translation or memory errors during the HTW operation. As a result of
which, HTW must not be considered a complete replacement for the TLB
refill software handler, but rather a fast-path for it.
For HTW to work, the PWBase register must contain the task's page
global directory address so the HTW will kick in on TLB refill
exceptions.

Due to HTW being a separate engine embedded deep in the CPU pipeline,
we need to restart the HTW everytime a PTE changes to avoid HTW
fetching a old entry from the page tables. It's also necessary to
restart the HTW on context switches to prevent it from fetching a
page from the previous process. Finally, since HTW is using the
entryhi register to write the translations to the TLB, it's necessary
to stop the HTW whenever the entryhi changes (eg for tlb probe
perations) and enable it back afterwards.

== Performance ==

The following trivial test was used to measure the performance of the
HTW. Using the same root filesystem, the following command was used
to measure the number of tlb refill handler executions with and
without (using 'nohtw' kernel parameter) HTW support.  The kernel was
modified to use a scratch register as a counter for the TLB refill
exceptions.

find /usr -type f -exec ls -lh {} \;

HTW Enabled:
TLB refill exceptions: 12306

HTW Disabled:
TLB refill exceptions: 17805

Signed-off-by: Markos Chandras 
Cc: linux-mips@linux-mips.org
Cc: Markos Chandras 
Patchwork: https://patchwork.linux-mips.org/patch/7336/
Signed-off-by: Ralf Baechle

MIPS: add kmap_noncoherent to wire a cached non-coherent TLB entry

2014-05-28T15:20:14+00:00

This is identical to kmap_coherent apart from the cache coherency
attribute used for the TLB entry, so kmap_coherent is abstracted to
kmap_prot which is then called for both kmap_coherent &
kmap_noncoherent. This will be used by a subsequent patch.

Suggested-by: Leonid Yegoshin 
Signed-off-by: Paul Burton