linux-stable.git/mm/vmalloc.c, branch v3.12.52

vmalloc: use rcu list iterator to reduce vmap_area_lock contention

2014-09-26T09:52:07+00:00

commit 474750aba88817c53f39424e5567b8e4acc4b39b upstream.

Richard Yao reported a month ago that his system have a trouble with
vmap_area_lock contention during performance analysis by /proc/meminfo.
Andrew asked why his analysis checks /proc/meminfo stressfully, but he
didn't answer it.

  https://lkml.org/lkml/2014/4/10/416

Although I'm not sure that this is right usage or not, there is a
solution reducing vmap_area_lock contention with no side-effect.  That
is just to use rcu list iterator in get_vmalloc_info().

rcu can be used in this function because all RCU protocol is already
respected by writers, since Nick Piggin commit db64fe02258f1 ("mm:
rewrite vmap layer") back in linux-2.6.28

Specifically :
   insertions use list_add_rcu(),
   deletions use list_del_rcu() and kfree_rcu().

Note the rb tree is not used from rcu reader (it would not be safe),
only the vmap_area_list has full RCU protection.

Note that __purge_vmap_area_lazy() already uses this rcu protection.

        rcu_read_lock();
        list_for_each_entry_rcu(va, &vmap_area_list, list) {
                if (va->flags & VM_LAZY_FREE) {
                        if (va->va_start < *start)
                                *start = va->va_start;
                        if (va->va_end > *end)
                                *end = va->va_end;
                        nr += (va->va_end - va->va_start) >> PAGE_SHIFT;
                        list_add_tail(&va->purge_list, &valist);
                        va->flags |= VM_LAZY_FREEING;
                        va->flags &= ~VM_LAZY_FREE;
                }
        }
        rcu_read_unlock();

Peter:

: While rcu list traversal over the vmap_area_list is safe, this may
: arrive at different results than the spinlocked version. The rcu list
: traversal version will not be a 'snapshot' of a single, valid instant
: of the entire vmap_area_list, but rather a potential amalgam of
: different list states.

Joonsoo:

: Yes, you are right, but I don't think that we should be strict here.
: Meminfo is already not a 'snapshot' at specific time.  While we try to get
: certain stats, the other stats can change.  And, although we may arrive at
: different results than the spinlocked version, the difference would not be
: large and would not make serious side-effect.

[edumazet@google.com: add more commit description]
Signed-off-by: Joonsoo Kim 
Reported-by: Richard Yao 
Acked-by: Eric Dumazet 
Cc: Peter Hurley 
Cc: Zhang Yanfei 
Cc: Johannes Weiner 
Cc: Andi Kleen 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds 
Signed-off-by: Mel Gorman 
Signed-off-by: Jiri Slaby

mm: kmemleak: avoid false negatives on vmalloc'ed objects

2014-07-28T07:12:23+00:00

commit 7f88f88f83ed609650a01b18572e605ea50cd163 upstream.

Commit 248ac0e1943a ("mm/vmalloc: remove guard page from between vmap
blocks") had the side effect of making vmap_area.va_end member point to
the next vmap_area.va_start.  This was creating an artificial reference
to vmalloc'ed objects and kmemleak was rarely reporting vmalloc() leaks.

This patch marks the vmap_area containing pointers explicitly and
reduces the min ref_count to 2 as vm_struct still contains a reference
to the vmalloc'ed object.  The kmemleak add_scan_area() function has
been improved to allow a SIZE_MAX argument covering the rest of the
object (for simpler calling sites).

Signed-off-by: Catalin Marinas 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds 
Signed-off-by: Jiri Slaby

mm/vmalloc: use wrapper function get_vm_area_size to caculate size of vm area

2013-09-11T22:58:02+00:00

Use wrapper function get_vm_area_size to calculate size of vm area.

Signed-off-by: Wanpeng Li 
Cc: Dave Hansen 
Cc: Rik van Riel 
Cc: Fengguang Wu 
Cc: Joonsoo Kim 
Cc: Johannes Weiner 
Cc: Tejun Heo 
Cc: Yasuaki Ishimatsu 
Cc: David Rientjes 
Cc: KOSAKI Motohiro 
Cc: Jiri Kosina 
Cc: Wanpeng Li 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm, vmalloc: use well-defined find_last_bit() func

2013-09-11T22:57:34+00:00

Our intention in here is to find last_bit within the region to flush.
There is well-defined function, find_last_bit() for this purpose and its
performance may be slightly better than current implementation.  So change
it.

Signed-off-by: Joonsoo Kim 
Reviewed-by: Wanpeng Li 
Acked-by: Johannes Weiner 
Acked-by: Zhang Yanfei 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm, vmalloc: remove useless variable in vmap_block

2013-09-11T22:57:33+00:00

vbq in vmap_block isn't used. So remove it.

Signed-off-by: Joonsoo Kim 
Reviewed-by: Wanpeng Li 
Acked-by: Johannes Weiner 
Acked-by: Zhang Yanfei 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/vmalloc.c: fix an overflow bug in alloc_vmap_area()

2013-07-09T17:33:23+00:00

When searching a vmap area in the vmalloc space, we use (addr + size -
1) to check if the value is less than addr, which is an overflow.  But
we assign (addr + size) to vmap_area->va_end.

So if we come across the below case:

  (addr + size - 1) : not overflow
  (addr + size)     : overflow

we will assign an overflow value (e.g 0) to vmap_area->va_end, And this
will trigger BUG in __insert_vmap_area, causing system panic.

So using (addr + size) to check the overflow should be the correct
behaviour, not (addr + size - 1).

Signed-off-by: Zhang Yanfei 
Reported-by: Ghennadi Procopciuc 
Tested-by: Daniel Baluta 
Cc: David Rientjes 
Cc: Minchan Kim 
Cc: KOSAKI Motohiro 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

vfree: don't schedule free_work() if llist_add() returns false

2013-07-09T17:33:22+00:00

vfree() only needs schedule_work(&p->wq) if p->list was empty, otherwise
vfree_deferred->wq is already pending or it is running and didn't do
llist_del_all() yet.

Signed-off-by: Oleg Nesterov 
Cc: Al Viro 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/vmalloc.c: check VM_UNINITIALIZED flag in s_show instead of show_numa_info

2013-07-09T17:33:21+00:00

We should check the VM_UNITIALIZED flag in s_show().  If this flag is
set, that said, the vm_struct is not fully initialized.  So it is
unnecessary to try to show the information contained in vm_struct.

We checked this flag in show_numa_info(), but I think it's better to
check it earlier.

Signed-off-by: Zhang Yanfei 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/vmalloc.c: rename VM_UNLIST to VM_UNINITIALIZED

2013-07-09T17:33:21+00:00

VM_UNLIST was used to indicate that the vm_struct is not listed in
vmlist.

But after commit 4341fa454796 ("mm, vmalloc: remove list management of
vmlist after initializing vmalloc"), the meaning of this flag changed.
It now means the vm_struct is not fully initialized.  So renaming it to
VM_UNINITIALIZED seems more reasonable.

Also change clear_vm_unlist to clear_vm_uninitialized_flag.

Signed-off-by: Zhang Yanfei 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/vmalloc.c: emit the failure message before return

2013-07-09T17:33:21+00:00

Use goto to jump to the fail label to give a failure message before
returning NULL.  This makes the failure handling in this function
consistent.

Signed-off-by: Zhang Yanfei 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds