linux-stable.git/mm/rmap.c, branch v3.0.2

mm/memory-failure.c: fix spinlock vs mutex order

2011-06-28T01:00:13+00:00

We cannot take a mutex while holding a spinlock, so flip the order and
fix the locking documentation.

Signed-off-by: Peter Zijlstra 
Acked-by: Andi Kleen 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: avoid anon_vma_chain allocation under anon_vma lock

2011-06-18T02:24:11+00:00

Hugh Dickins points out that lockdep (correctly) spots a potential
deadlock on the anon_vma lock, because we now do a GFP_KERNEL allocation
of anon_vma_chain while doing anon_vma_clone().  The problem is that
page reclaim will want to take the anon_vma lock of any anonymous pages
that it will try to reclaim.

So re-organize the code in anon_vma_clone() slightly: first do just a
GFP_NOWAIT allocation, which will usually work fine.  But if that fails,
let's just drop the lock and re-do the allocation, now with GFP_KERNEL.

End result: not only do we avoid the locking problem, this also ends up
getting better concurrency in case the allocation does need to block.
Tim Chen reports that with all these anon_vma locking tweaks, we're now
almost back up to the spinlock performance.

Reported-and-tested-by: Hugh Dickins 
Tested-by: Tim Chen 
Cc: Peter Zijlstra 
Cc: Andi Kleen 
Signed-off-by: Linus Torvalds

mm: avoid repeated anon_vma lock/unlock sequences in unlink_anon_vmas()

2011-06-18T02:23:52+00:00

This matches the anon_vma_clone() case, and uses the same lock helper
functions.  Because of the need to potentially release the anon_vma's,
it's a bit more complex, though.

We traverse the 'vma->anon_vma_chain' in two phases: the first loop gets
the anon_vma lock (with the helper function that only takes the lock
once for the whole loop), and removes any entries that don't need any
more processing.

The second phase just traverses the remaining list entries (without
holding the anon_vma lock), and does any actual freeing of the
anon_vma's that is required.

Signed-off-by: Peter Zijlstra 
Tested-by: Hugh Dickins 
Tested-by: Tim Chen 
Cc: Andi Kleen 
Signed-off-by: Linus Torvalds

mm: avoid repeated anon_vma lock/unlock sequences in anon_vma_clone()

2011-06-18T02:20:49+00:00

In anon_vma_clone() we traverse the vma->anon_vma_chain of the source
vma, locking the anon_vma for each entry.

But they are all going to have the same root entry, which means that
we're locking and unlocking the same lock over and over again.  Which is
expensive in locked operations, but can get _really_ expensive when that
root entry sees any kind of lock contention.

In fact, Tim Chen reports a big performance regression due to this: when
we switched to use a mutex instead of a spinlock, the contention case
gets much worse.

So to alleviate this all, this commit creates a small helper function
(lock_anon_vma_root()) that can be used to take the lock just once
rather than taking and releasing it over and over again.

We still have the same "take the lock and release" it behavior in the
exit path (in unlink_anon_vmas()), but that one is a bit harder to fix
since we're actually freeing the anon_vma entries as we go, and that
will touch the lock too.

Reported-and-tested-by: Tim Chen 
Tested-by: Hugh Dickins 
Cc: Peter Zijlstra 
Cc: Andi Kleen 
Signed-off-by: Linus Torvalds

mm, rmap: Add yet more comments to page_get_anon_vma/page_lock_anon_vma

2011-05-29T16:25:48+00:00

Inspired by an analysis from Hugh on why again all this doesn't explode
in our face.

Signed-off-by: Peter Zijlstra 
Signed-off-by: Linus Torvalds

mm: fix page_lock_anon_vma leaving mutex locked

2011-05-28T23:55:32+00:00

On one machine I've been getting hangs, a page fault's anon_vma_prepare()
waiting in anon_vma_lock(), other processes waiting for that page's lock.

This is a replay of last year's f18194275c39 "mm: fix hang on
anon_vma->root->lock".

The new page_lock_anon_vma() places too much faith in its refcount: when
it has acquired the mutex_trylock(), it's possible that a racing task in
anon_vma_alloc() has just reallocated the struct anon_vma, set refcount
to 1, and is about to reset its anon_vma->root.

Fix this by saving anon_vma->root, and relying on the usual page_mapped()
check instead of a refcount check: if page is still mapped, the anon_vma
is still ours; if page is not still mapped, we're no longer interested.

Signed-off-by: Hugh Dickins 
Signed-off-by: Linus Torvalds

mm: fix kernel BUG at mm/rmap.c:1017!

2011-05-28T23:09:26+00:00

I've hit the "address >= vma->vm_end" check in do_page_add_anon_rmap()
just once.  The stack showed khugepaged allocation trying to compact
pages: the call to page_add_anon_rmap() coming from remove_migration_pte().

That path holds anon_vma lock, but does not hold mmap_sem: it can
therefore race with a split_vma(), and in commit 5f70b962ccc2 "mmap:
avoid unnecessary anon_vma lock" we just took away the anon_vma lock
protection when adjusting vma->vm_end.

I don't think that particular BUG_ON ever caught anything interesting,
so better replace it by a comment, than reinstate the anon_vma locking.

Signed-off-by: Hugh Dickins 
Signed-off-by: Linus Torvalds

mm: optimize page_lock_anon_vma() fast-path

2011-05-25T15:39:20+00:00

Optimize the page_lock_anon_vma() fast path to be one atomic op, instead
of two.

Signed-off-by: Peter Zijlstra 
Reviewed-by: KAMEZAWA Hiroyuki 
Cc: Benjamin Herrenschmidt 
Cc: David Miller 
Cc: Martin Schwidefsky 
Cc: Russell King 
Cc: Paul Mundt 
Cc: Jeff Dike 
Cc: Richard Weinberger 
Cc: Tony Luck 
Cc: Hugh Dickins 
Cc: Mel Gorman 
Cc: KOSAKI Motohiro 
Cc: Nick Piggin 
Cc: Namhyung Kim 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: convert anon_vma->lock to a mutex

2011-05-25T15:39:19+00:00

Straightforward conversion of anon_vma->lock to a mutex.

Signed-off-by: Peter Zijlstra 
Acked-by: Hugh Dickins 
Reviewed-by: KOSAKI Motohiro 
Cc: Benjamin Herrenschmidt 
Cc: David Miller 
Cc: Martin Schwidefsky 
Cc: Russell King 
Cc: Paul Mundt 
Cc: Jeff Dike 
Cc: Richard Weinberger 
Cc: Tony Luck 
Cc: KAMEZAWA Hiroyuki 
Cc: Mel Gorman 
Cc: Nick Piggin 
Cc: Namhyung Kim 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: use refcounts for page_lock_anon_vma()

2011-05-25T15:39:19+00:00

Convert page_lock_anon_vma() over to use refcounts.  This is done to
prepare for the conversion of anon_vma from spinlock to mutex.

Sadly this inceases the cost of page_lock_anon_vma() from one to two
atomics, a follow up patch addresses this, lets keep that simple for now.

Signed-off-by: Peter Zijlstra 
Reviewed-by: KAMEZAWA Hiroyuki 
Reviewed-by: KOSAKI Motohiro 
Acked-by: Hugh Dickins 
Cc: Benjamin Herrenschmidt 
Cc: David Miller 
Cc: Martin Schwidefsky 
Cc: Russell King 
Cc: Paul Mundt 
Cc: Jeff Dike 
Cc: Richard Weinberger 
Cc: Tony Luck 
Cc: Mel Gorman 
Cc: Nick Piggin 
Cc: Namhyung Kim 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds