linux-stable.git/arch/mips/include/asm/pgtable.h, branch linux-4.5.y

MIPS: Sync icache & dcache in set_pte_at

2016-06-08T01:18:46+00:00

commit 37d22a0d798b5c938b277d32cfd86dc231381342 upstream.

It's possible for pages to become visible prior to update_mmu_cache
running if a thread within the same address space preempts the current
thread or runs simultaneously on another CPU. That is, the following
scenario is possible:

    CPU0                            CPU1

    write to page
    flush_dcache_page
    flush_icache_page
    set_pte_at
                                    map page
    update_mmu_cache

If CPU1 maps the page in between CPU0's set_pte_at, which marks it valid
& visible, and update_mmu_cache where the dcache flush occurs then CPU1s
icache will fill from stale data (unless it fills from the dcache, in
which case all is good, but most MIPS CPUs don't have this property).
Commit 4d46a67a3eb8 ("MIPS: Fix race condition in lazy cache flushing.")
attempted to fix that by performing the dcache flush in
flush_icache_page such that it occurs before the set_pte_at call makes
the page visible. However it has the problem that not all code that
writes to pages exposed to userland call flush_icache_page. There are
many callers of set_pte_at under mm/ and only 2 of them do call
flush_icache_page. Thus the race window between a page becoming visible
& being coherent between the icache & dcache remains open in some cases.

To illustrate some of the cases, a WARN was added to __update_cache with
this patch applied that triggered in cases where a page about to be
flushed from the dcache was not the last page provided to
flush_icache_page. That is, backtraces were obtained for cases in which
the race window is left open without this patch. The 2 standout examples
follow.

When forking a process:

[   15.271842] [<80417630>] __update_cache+0xcc/0x188
[   15.277274] [<80530394>] copy_page_range+0x56c/0x6ac
[   15.282861] [<8042936c>] copy_process.part.54+0xd40/0x17ac
[   15.289028] [<80429f80>] do_fork+0xe4/0x420
[   15.293747] [<80413808>] handle_sys+0x128/0x14c

When exec'ing an ELF binary:

[   14.445964] [<80417630>] __update_cache+0xcc/0x188
[   14.451369] [<80538d88>] move_page_tables+0x414/0x498
[   14.457075] [<8055d848>] setup_arg_pages+0x220/0x318
[   14.462685] [<805b0f38>] load_elf_binary+0x530/0x12a0
[   14.468374] [<8055ec3c>] search_binary_handler+0xbc/0x214
[   14.474444] [<8055f6c0>] do_execveat_common+0x43c/0x67c
[   14.480324] [<8055f938>] do_execve+0x38/0x44
[   14.485137] [<80413808>] handle_sys+0x128/0x14c

These code paths write into a page, call flush_dcache_page then call
set_pte_at without flush_icache_page inbetween. The end result is that
the icache can become corrupted & userland processes may execute
unexpected or invalid code, typically resulting in a reserved
instruction exception, a trap or a segfault.

Fix this race condition fully by performing any cache maintenance
required to keep the icache & dcache in sync in set_pte_at, before the
page is made valid. This has the added bonus of ensuring the cache
maintenance always happens in one location, rather than being duplicated
in flush_icache_page & update_mmu_cache. It also matches the way other
architectures solve the same problem (see arm, ia64 & powerpc).

Signed-off-by: Paul Burton 
Reported-by: Ionela Voinescu 
Cc: Lars Persson 
Fixes: 4d46a67a3eb8 ("MIPS: Fix race condition in lazy cache flushing.")
Cc: Steven J. Hill 
Cc: David Daney 
Cc: Huacai Chen 
Cc: Aneesh Kumar K.V 
Cc: Andrew Morton 
Cc: Jerome Marchand 
Cc: Kirill A. Shutemov 
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12722/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus

2016-01-24T20:50:56+00:00

Pull MIPS updates from Ralf Baechle:
 "This is the main pull request for MIPS for 4.5 plus some 4.4 fixes.

  The executive summary:

   - ATH79 platform improvments, use DT bindings for the ATH79 USB PHY.
   - Avoid useless rebuilds for zboot.
   - jz4780: Add NEMC, BCH and NAND device tree nodes
   - Initial support for the MicroChip's DT platform.  As all the device
     drivers are missing this is still of limited use.
   - Some Loongson3 cleanups.
   - The unavoidable whitespace polishing.
   - Reduce clock skew when synchronizing the CPU cycle counters on CPU
     startup.
   - Add MIPS R6 fixes.
   - Lots of cleanups across arch/mips as fallout from KVM.
   - Lots of minor fixes and changes for IEEE 754-2008 support to the
     FPU emulator / fp-assist software.
   - Minor Ralink, BCM47xx and bcm963xx platform support improvments.
   - Support SMP on BCM63168"

* 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus: (84 commits)
  MIPS: zboot: Add support for serial debug using the PROM
  MIPS: zboot: Avoid useless rebuilds
  MIPS: BMIPS: Enable ARCH_WANT_OPTIONAL_GPIOLIB
  MIPS: bcm63xx: nvram: Remove unused bcm63xx_nvram_get_psi_size() function
  MIPS: bcm963xx: Update bcm_tag field image_sequence
  MIPS: bcm963xx: Move extended flash address to bcm_tag header file
  MIPS: bcm963xx: Move Broadcom BCM963xx image tag data structure
  MIPS: bcm63xx: nvram: Use nvram structure definition from header file
  MIPS: bcm963xx: Add Broadcom BCM963xx board nvram data structure
  MAINTAINERS: Add KVM for MIPS entry
  MIPS: KVM: Add missing newline to kvm_err()
  MIPS: Move KVM specific opcodes into asm/inst.h
  MIPS: KVM: Use cacheops.h definitions
  MIPS: Break down cacheops.h definitions
  MIPS: Use EXCCODE_ constants with set_except_vector()
  MIPS: Update trap codes
  MIPS: Move Cause.ExcCode trap codes to mipsregs.h
  MIPS: KVM: Make kvm_mips_{init,exit}() static
  MIPS: KVM: Refactor added offsetof()s
  MIPS: KVM: Convert EXPORT_SYMBOL to _GPL
  ...

MIPS: Fix some missing CONFIG_CPU_MIPSR6 #ifdefs

2016-01-24T00:32:25+00:00

Commit be0c37c985eddc4 (MIPS: Rearrange PTE bits into fixed positions.)
defines fixed PTE bits for MIPS R2. Then, commit d7b631419b3d230a4d383
(MIPS: pgtable-bits: Fix XPA damage to R6 definitions.) adds the MIPS
R6 definitions in the same way as MIPS R2. But some R6 #ifdefs in the
later commit are missing, so in this patch I fix that.

Signed-off-by: Huacai Chen 
Cc: Aurelien Jarno 
Cc: Steven J. Hill 
Cc: Fuxin Zhang 
Cc: Zhangjin Wu 
Cc: linux-mips@linux-mips.org
Cc: stable@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12164/
Signed-off-by: Ralf Baechle

mips, thp: remove infrastructure for handling splitting PMDs

2016-01-16T01:56:32+00:00

With new refcounting we don't need to mark PMDs splitting.  Let's drop
code to handle this.

pmdp_splitting_flush() is not needed too: on splitting PMD we will do
pmdp_clear_flush() + set_pte_at().  pmdp_clear_flush() will do IPI as
needed for fast_gup.

Signed-off-by: Kirill A. Shutemov 
Cc: Sasha Levin 
Cc: Aneesh Kumar K.V 
Cc: Jerome Marchand 
Cc: Vlastimil Babka 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Christoph Lameter 
Cc: David Rientjes 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

MIPS: Fix definition of pgprot_writecombine()

2015-09-03T10:08:00+00:00

If pgprot_writecombine is not #defined, asm-generic/pgtable.h will try
to provide a default implementation by #defining it to pgprot_noncached.
However our implementation is an inline function rather than a #define,
so it was never actually used because of the #define in generic code.

Add "#define pgprot_writecombine pgprot_writecombine" to prevent generic
code from re-defining it.

Signed-off-by: Alex Smith 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10767/
Signed-off-by: Ralf Baechle

MIPS: Make set_pte() SMP safe.

2015-08-05T09:11:10+00:00

On MIPS the GLOBAL bit of the PTE must have the same value in any
aligned pair of PTEs.  These pairs of PTEs are referred to as
"buddies".  In a SMP system is is possible for two CPUs to be calling
set_pte() on adjacent PTEs at the same time.  There is a race between
setting the PTE and a different CPU setting the GLOBAL bit in its
buddy PTE.

This race can be observed when multiple CPUs are executing
vmap()/vfree() at the same time.

Make setting the buddy PTE's GLOBAL bit an atomic operation to close
the race condition.

The case of CONFIG_64BIT_PHYS_ADDR && CONFIG_CPU_MIPS32 is *not*
handled.

Signed-off-by: David Daney 
Cc: 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10835/
Signed-off-by: Ralf Baechle

mm: clarify that the function operates on hugepage pte

2015-06-25T00:49:44+00:00

We have confusing functions to clear pmd, pmd_clear_* and pmd_clear.  Add
_huge_ to pmdp_clear functions so that we are clear that they operate on
hugepage pte.

We don't bother about other functions like pmdp_set_wrprotect,
pmdp_clear_flush_young, because they operate on PTE bits and hence
indicate they are operating on hugepage ptes

Signed-off-by: Aneesh Kumar K.V 
Acked-by: Kirill A. Shutemov 
Cc: Benjamin Herrenschmidt 
Cc: Paul Mackerras 
Cc: Michael Ellerman 
Cc: Andrea Arcangeli 
Cc: Martin Schwidefsky 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

Revert "MIPS: Remove race window in page fault handling"

2015-03-25T12:48:00+00:00

Revert commit 2a4a8b1e5d9d ("MIPS: Remove race window in page fault
handling") because it increased the number of flushed dcache pages and
became a performance problem for some workloads.

Signed-off-by: Lars Persson 
Cc: linux-mips@linux-mips.org
Cc: paul.burton@imgtec.com
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/9345/
Signed-off-by: Ralf Baechle

MIPS: Add support for XPA.

2015-03-19T16:39:49+00:00

Add support for extended physical addressing (XPA) so that
32-bit platforms can access equal to or greater than 40 bits
of physical addresses.

NOTE:
      1) XPA and EVA are not the same and cannot be used
         simultaneously.
      2) If you configure your kernel for XPA, the PTEs
         and all address sizes become 64-bit.
      3) Your platform MUST have working HIGHMEM support.

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9355/
Signed-off-by: Ralf Baechle

MIPS: Rearrange PTE bits into fixed positions.

2015-03-18T15:19:35+00:00

This patch rearranges the PTE bits into fixed positions for R2
and later cores. In the past, the TLB handling code did runtime
checking of RI/XI and adjusted the shifts and rotates in order
to fit the largest PFN value into the PTE. The checking now
occurs when building the TLB handler, thus eliminating those
checks. These new arrangements also define the largest possible
PFN value that can fit in the PTE. HUGE page support is only
available for 64-bit cores. Layouts of the PTE bits are now:

   64-bit, R1 or earlier:     CCC D V G [S H] M A W R P
   32-bit, R1 or earler:      CCC D V G M A W R P
   64-bit, R2 or later:       CCC D V G RI/R XI [S H] M A W P
   32-bit, R2 or later:       CCC D V G RI/R XI M A W P

[ralf@linux-mips.org: Fix another build error *rant* *rant*]

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9353/
Signed-off-by: Ralf Baechle