linux.git/mm, branch v4.13

Merge branch 'akpm' (patches from Andrew)

2017-09-01T00:56:56+00:00

Merge more fixes from Andrew Morton:
 "6 fixes"

* emailed patches from Andrew Morton :
  scripts/dtc: fix '%zx' warning
  include/linux/compiler.h: don't perform compiletime_assert with -O0
  mm, madvise: ensure poisoned pages are removed from per-cpu lists
  mm, uprobes: fix multiple free of ->uprobes_state.xol_area
  kernel/kthread.c: kthread_worker: don't hog the cpu
  mm,page_alloc: don't call __node_reclaim() with oom_lock held.

mm, madvise: ensure poisoned pages are removed from per-cpu lists

2017-08-31T23:33:15+00:00

Wendy Wang reported off-list that a RAS HWPOISON-SOFT test case failed
and bisected it to the commit 479f854a207c ("mm, page_alloc: defer
debugging checks of pages allocated from the PCP").

The problem is that a page that was poisoned with madvise() is reused.
The commit removed a check that would trigger if DEBUG_VM was enabled
but re-enabling the check only fixes the problem as a side-effect by
printing a bad_page warning and recovering.

The root of the problem is that an madvise() can leave a poisoned page
on the per-cpu list.  This patch drains all per-cpu lists after pages
are poisoned so that they will not be reused.  Wendy reports that the
test case in question passes with this patch applied.  While this could
be done in a targeted fashion, it is over-complicated for such a rare
operation.

Link: http://lkml.kernel.org/r/20170828133414.7qro57jbepdcyz5x@techsingularity.net
Fixes: 479f854a207c ("mm, page_alloc: defer debugging checks of pages allocated from the PCP")
Signed-off-by: Mel Gorman 
Reported-by: Wang, Wendy 
Tested-by: Wang, Wendy 
Acked-by: David Rientjes 
Acked-by: Vlastimil Babka 
Cc: "Hansen, Dave" 
Cc: "Luck, Tony" 
Cc: Naoya Horiguchi 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm,page_alloc: don't call __node_reclaim() with oom_lock held.

2017-08-31T23:33:14+00:00

We are doing a last second memory allocation attempt before calling
out_of_memory().  But since slab shrinker functions might indirectly
wait for other thread's __GFP_DIRECT_RECLAIM && !__GFP_NORETRY memory
allocations via sleeping locks, calling slab shrinker functions from
node_reclaim() from get_page_from_freelist() with oom_lock held has
possibility of deadlock.  Therefore, make sure that last second memory
allocation attempt does not call slab shrinker functions.

Link: http://lkml.kernel.org/r/1503577106-9196-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
Signed-off-by: Tetsuo Handa 
Acked-by: Michal Hocko 
Cc: Mel Gorman 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/mmu_notifier: kill invalidate_page

2017-08-31T23:13:00+00:00

The invalidate_page callback suffered from two pitfalls.  First it used
to happen after the page table lock was release and thus a new page
might have setup before the call to invalidate_page() happened.

This is in a weird way fixed by commit c7ab0d2fdc84 ("mm: convert
try_to_unmap_one() to use page_vma_mapped_walk()") that moved the
callback under the page table lock but this also broke several existing
users of the mmu_notifier API that assumed they could sleep inside this
callback.

The second pitfall was invalidate_page() being the only callback not
taking a range of address in respect to invalidation but was giving an
address and a page.  Lots of the callback implementers assumed this
could never be THP and thus failed to invalidate the appropriate range
for THP.

By killing this callback we unify the mmu_notifier callback API to
always take a virtual address range as input.

Finally this also simplifies the end user life as there is now two clear
choices:
  - invalidate_range_start()/end() callback (which allow you to sleep)
  - invalidate_range() where you can not sleep but happen right after
    page table update under page table lock

Signed-off-by: Jérôme Glisse 
Cc: Bernhard Held 
Cc: Adam Borowski 
Cc: Andrea Arcangeli 
Cc: Radim Krčmář 
Cc: Wanpeng Li 
Cc: Paolo Bonzini 
Cc: Takashi Iwai 
Cc: Nadav Amit 
Cc: Mike Galbraith 
Cc: Kirill A. Shutemov 
Cc: axie 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/rmap: update to new mmu_notifier semantic v2

2017-08-31T23:12:59+00:00

Replace all mmu_notifier_invalidate_page() calls by *_invalidate_range()
and make sure it is bracketed by calls to *_invalidate_range_start()/end().

Note that because we can not presume the pmd value or pte value we have
to assume the worst and unconditionaly report an invalidation as
happening.

Changed since v2:
  - try_to_unmap_one() only one call to mmu_notifier_invalidate_range()
  - compute end with PAGE_SIZE << compound_order(page)
  - fix PageHuge() case in try_to_unmap_one()

Signed-off-by: Jérôme Glisse 
Reviewed-by: Andrea Arcangeli 
Cc: Dan Williams 
Cc: Ross Zwisler 
Cc: Bernhard Held 
Cc: Adam Borowski 
Cc: Radim Krčmář 
Cc: Wanpeng Li 
Cc: Paolo Bonzini 
Cc: Takashi Iwai 
Cc: Nadav Amit 
Cc: Mike Galbraith 
Cc: Kirill A. Shutemov 
Cc: axie 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds

dax: update to new mmu_notifier semantic

2017-08-31T23:12:59+00:00

Replace all mmu_notifier_invalidate_page() calls by *_invalidate_range()
and make sure it is bracketed by calls to *_invalidate_range_start()/end().

Note that because we can not presume the pmd value or pte value we have
to assume the worst and unconditionaly report an invalidation as
happening.

Signed-off-by: Jérôme Glisse 
Cc: Dan Williams 
Cc: Ross Zwisler 
Cc: Bernhard Held 
Cc: Adam Borowski 
Cc: Andrea Arcangeli 
Cc: Radim Krčmář 
Cc: Wanpeng Li 
Cc: Paolo Bonzini 
Cc: Takashi Iwai 
Cc: Nadav Amit 
Cc: Mike Galbraith 
Cc: Kirill A. Shutemov 
Cc: axie 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds

Revert "rmap: do not call mmu_notifier_invalidate_page() under ptl"

2017-08-29T16:11:06+00:00

This reverts commit aac2fea94f7a3df8ad1eeb477eb2643f81fd5393.

It turns out that that patch was complete and utter garbage, and broke
KVM, resulting in odd oopses.

Quoting Andrea Arcangeli:
 "The aforementioned commit has 3 bugs.

  1) mmu_notifier_invalidate_range cannot be used in replacement of
     mmu_notifier_invalidate_range_start/end.

     For KVM mmu_notifier_invalidate_range is a noop and rightfully so.

     A MMU notifier implementation has to implement either
     ->invalidate_range method or the invalidate_range_start/end
     methods, not both. And if you implement invalidate_range_start/end
     like KVM is forced to do, calling mmu_notifier_invalidate_range in
     common code is a noop for KVM.

     For those MMU notifiers that can get away only implementing
     ->invalidate_range, the ->invalidate_range is implicitly called by
     mmu_notifier_invalidate_range_end(). And only those secondary MMUs
     that share the same pagetable with the primary MMU (like AMD
     iommuv2) can get away only implementing ->invalidate_range.

     So all cases (THP on/off) are broken right now.

     To fix this is enough to replace mmu_notifier_invalidate_range with
     mmu_notifier_invalidate_range_start;mmu_notifier_invalidate_range_end.
     Either that or call multiple mmu_notifier_invalidate_page like
     before.

  2) address + (1UL << compound_order(page) is buggy, it should be
     PAGE_SIZE << compound_order(page), it's bytes not pages, 2M not
     512.

  3) The whole invalidate_range thing was an attempt to call a single
     invalidate while walking multiple 4k ptes that maps the same THP
     (after a pmd virtual split without physical compound page THP
     split).

     It's unclear if the rmap_walk will always provide an address that
     is 2M aligned as parameter to try_to_unmap_one, in presence of THP.
     I think it needs also an address &= (PAGE_SIZE <<
     compound_order(page)) - 1 to be safe"

In general, we should stop making excuses for horrible MMU notifier
users.  It's much more important that the core VM is sane and safe, than
letting MMU notifiers sleep.

So if some MMU notifier is sleeping under a spinlock, we need to fix the
notifier, not try to make excuses for that garbage in the core VM.

Reported-and-tested-by: Bernhard Held 
Reported-and-tested-by: Adam Borowski 
Cc: Andrea Arcangeli 
Cc: Radim Krčmář 
Cc: Wanpeng Li 
Cc: Paolo Bonzini 
Cc: Takashi Iwai 
Cc: Nadav Amit 
Cc: Mike Galbraith 
Cc: Kirill A. Shutemov 
Cc: Jérôme Glisse 
Cc: axie 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds

page waitqueue: always add new entries at the end

2017-08-28T23:45:40+00:00

Commit 3510ca20ece0 ("Minor page waitqueue cleanups") made the page
queue code always add new waiters to the back of the queue, which helps
upcoming patches to batch the wakeups for some horrid loads where the
wait queues grow to thousands of entries.

However, I forgot about the nasrt add_page_wait_queue() special case
code that is only used by the cachefiles code.  That one still continued
to add the new wait queue entries at the beginning of the list.

Fix it, because any sane batched wakeup will require that we don't
suddenly start getting new entries at the beginning of the list that we
already handled in a previous batch.

[ The current code always does the whole list while holding the lock, so
  wait queue ordering doesn't matter for correctness, but even then it's
  better to add later entries at the end from a fairness standpoint ]

Signed-off-by: Linus Torvalds

Avoid page waitqueue race leaving possible page locker waiting

2017-08-27T23:25:09+00:00

The "lock_page_killable()" function waits for exclusive access to the
page lock bit using the WQ_FLAG_EXCLUSIVE bit in the waitqueue entry
set.

That means that if it gets woken up, other waiters may have been
skipped.

That, in turn, means that if it sees the page being unlocked, it *must*
take that lock and return success, even if a lethal signal is also
pending.

So instead of checking for lethal signals first, we need to check for
them after we've checked the actual bit that we were waiting for.  Even
if that might then delay the killing of the process.

This matches the order of the old "wait_on_bit_lock()" infrastructure
that the page locking used to use (and is still used in a few other
areas).

Note that if we still return an error after having unsuccessfully tried
to acquire the page lock, that is ok: that means that some other thread
was able to get ahead of us and lock the page, and when that other
thread then unlocks the page, the wakeup event will be repeated.  So any
other pending waiters will now get properly woken up.

Fixes: 62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit")
Cc: Nick Piggin 
Cc: Peter Zijlstra 
Cc: Mel Gorman 
Cc: Jan Kara 
Cc: Davidlohr Bueso 
Cc: Andi Kleen 
Signed-off-by: Linus Torvalds

Minor page waitqueue cleanups

2017-08-27T20:55:12+00:00

Tim Chen and Kan Liang have been battling a customer load that shows
extremely long page wakeup lists.  The cause seems to be constant NUMA
migration of a hot page that is shared across a lot of threads, but the
actual root cause for the exact behavior has not been found.

Tim has a patch that batches the wait list traversal at wakeup time, so
that we at least don't get long uninterruptible cases where we traverse
and wake up thousands of processes and get nasty latency spikes.  That
is likely 4.14 material, but we're still discussing the page waitqueue
specific parts of it.

In the meantime, I've tried to look at making the page wait queues less
expensive, and failing miserably.  If you have thousands of threads
waiting for the same page, it will be painful.  We'll need to try to
figure out the NUMA balancing issue some day, in addition to avoiding
the excessive spinlock hold times.

That said, having tried to rewrite the page wait queues, I can at least
fix up some of the braindamage in the current situation. In particular:

 (a) we don't want to continue walking the page wait list if the bit
     we're waiting for already got set again (which seems to be one of
     the patterns of the bad load).  That makes no progress and just
     causes pointless cache pollution chasing the pointers.

 (b) we don't want to put the non-locking waiters always on the front of
     the queue, and the locking waiters always on the back.  Not only is
     that unfair, it means that we wake up thousands of reading threads
     that will just end up being blocked by the writer later anyway.

Also add a comment about the layout of 'struct wait_page_key' - there is
an external user of it in the cachefiles code that means that it has to
match the layout of 'struct wait_bit_key' in the two first members.  It
so happens to match, because 'struct page *' and 'unsigned long *' end
up having the same values simply because the page flags are the first
member in struct page.

Cc: Tim Chen 
Cc: Kan Liang 
Cc: Mel Gorman 
Cc: Christopher Lameter 
Cc: Andi Kleen 
Cc: Davidlohr Bueso 
Cc: Peter Zijlstra 
Signed-off-by: Linus Torvalds