linux.git/mm/sparse-vmemmap.c, branch v4.19

mm/sparse: delete old sparse_init and enable new one

2018-08-17T23:20:32+00:00

Rename new_sparse_init() to sparse_init() which enables it.  Delete old
sparse_init() and all the code that became obsolete with.

[pasha.tatashin@oracle.com: remove unused sparse_mem_maps_populate_node()]
  Link: http://lkml.kernel.org/r/20180716174447.14529-6-pasha.tatashin@oracle.com
Link: http://lkml.kernel.org/r/20180712203730.8703-6-pasha.tatashin@oracle.com
Signed-off-by: Pavel Tatashin 
Tested-by: Michael Ellerman 	[powerpc]
Tested-by: Oscar Salvador 
Reviewed-by: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Abdul Haleem 
Cc: Baoquan He 
Cc: Daniel Jordan 
Cc: Dan Williams 
Cc: Dave Hansen 
Cc: David Rientjes 
Cc: Greg Kroah-Hartman 
Cc: Ingo Molnar 
Cc: Jan Kara 
Cc: Jérôme Glisse 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Souptick Joarder 
Cc: Steven Sistare 
Cc: Vlastimil Babka 
Cc: Wei Yang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/sparse: move buffer init/fini to the common place

2018-08-17T23:20:32+00:00

Now that both variants of sparse memory use the same buffers to populate
memory map, we can move sparse_buffer_init()/sparse_buffer_fini() to the
common place.

Link: http://lkml.kernel.org/r/20180712203730.8703-4-pasha.tatashin@oracle.com
Signed-off-by: Pavel Tatashin 
Tested-by: Michael Ellerman 	[powerpc]
Tested-by: Oscar Salvador 
Reviewed-by: Andrew Morton 
Cc: Pasha Tatashin 
Cc: Abdul Haleem 
Cc: Baoquan He 
Cc: Daniel Jordan 
Cc: Dan Williams 
Cc: Dave Hansen 
Cc: David Rientjes 
Cc: Greg Kroah-Hartman 
Cc: Ingo Molnar 
Cc: Jan Kara 
Cc: Jérôme Glisse 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Souptick Joarder 
Cc: Steven Sistare 
Cc: Vlastimil Babka 
Cc: Wei Yang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/sparse: abstract sparse buffer allocations

2018-08-17T23:20:32+00:00

Patch series "sparse_init rewrite", v6.

In sparse_init() we allocate two large buffers to temporary hold usemap
and memmap for the whole machine.  However, we can avoid doing that if
we changed sparse_init() to operated on per-node bases instead of doing
it on the whole machine beforehand.

As shown by Baoquan
  http://lkml.kernel.org/r/20180628062857.29658-1-bhe@redhat.com

The buffers are large enough to cause machine stop to boot on small
memory systems.

Another benefit of these changes is that they also obsolete
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER.

This patch (of 5):

When struct pages are allocated for sparse-vmemmap VA layout, we first try
to allocate one large buffer, and than if that fails allocate struct pages
for each section as we go.

The code that allocates buffer is uses global variables and is spread
across several call sites.

Cleanup the code by introducing three functions to handle the global
buffer:

sparse_buffer_init()	initialize the buffer
sparse_buffer_fini()	free the remaining part of the buffer
sparse_buffer_alloc()	alloc from the buffer, and if buffer is empty
return NULL

Define these functions in sparse.c instead of sparse-vmemmap.c because
later we will use them for non-vmemmap sparse allocations as well.

[akpm@linux-foundation.org: use PTR_ALIGN()]
[akpm@linux-foundation.org: s/BUG_ON/WARN_ON/]
Link: http://lkml.kernel.org/r/20180712203730.8703-2-pasha.tatashin@oracle.com
Signed-off-by: Pavel Tatashin 
Tested-by: Michael Ellerman 	[powerpc]
Reviewed-by: Oscar Salvador 
Tested-by: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Steven Sistare 
Cc: Daniel Jordan 
Cc: "Kirill A. Shutemov" 
Cc: Michal Hocko 
Cc: Dan Williams 
Cc: Jan Kara 
Cc: Jérôme Glisse 
Cc: Souptick Joarder 
Cc: Baoquan He 
Cc: Greg Kroah-Hartman 
Cc: Vlastimil Babka 
Cc: Wei Yang 
Cc: Dave Hansen 
Cc: David Rientjes 
Cc: Ingo Molnar 
Cc: Abdul Haleem 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/sparse: optimize memmap allocation during sparse_init()

2018-08-17T23:20:31+00:00

In sparse_init(), two temporary pointer arrays, usemap_map and map_map
are allocated with the size of NR_MEM_SECTIONS.  They are used to store
each memory section's usemap and mem map if marked as present.  With the
help of these two arrays, continuous memory chunk is allocated for
usemap and memmap for memory sections on one node.  This avoids too many
memory fragmentations.  Like below diagram, '1' indicates the present
memory section, '0' means absent one.  The number 'n' could be much
smaller than NR_MEM_SECTIONS on most of systems.

  |1|1|1|1|0|0|0|0|1|1|0|0|...|1|0||1|0|...|1||0|1|...|0|
  -------------------------------------------------------
   0 1 2 3         4 5         i   i+1     n-1   n

If we fail to populate the page tables to map one section's memmap, its
->section_mem_map will be cleared finally to indicate that it's not
present.  After use, these two arrays will be released at the end of
sparse_init().

In 4-level paging mode, each array costs 4M which can be ignorable.
While in 5-level paging, they costs 256M each, 512M altogether.  Kdump
kernel Usually only reserves very few memory, e.g 256M.  So, even thouth
they are temporarily allocated, still not acceptable.

In fact, there's no need to allocate them with the size of
NR_MEM_SECTIONS.  Since the ->section_mem_map clearing has been deferred
to the last, the number of present memory sections are kept the same
during sparse_init() until we finally clear out the memory section's
->section_mem_map if its usemap or memmap is not correctly handled.
Thus in the middle whenever for_each_present_section_nr() loop is taken,
the i-th present memory section is always the same one.

Here only allocate usemap_map and map_map with the size of
'nr_present_sections'.  For the i-th present memory section, install its
usemap and memmap to usemap_map[i] and mam_map[i] during allocation.
Then in the last for_each_present_section_nr() loop which clears the
failed memory section's ->section_mem_map, fetch usemap and memmap from
usemap_map[] and map_map[] array and set them into mem_section[]
accordingly.

[akpm@linux-foundation.org: coding-style fixes]
Link: http://lkml.kernel.org/r/20180628062857.29658-5-bhe@redhat.com
Signed-off-by: Baoquan He 
Reviewed-by: Pavel Tatashin 
Cc: Pasha Tatashin 
Cc: Dave Hansen 
Cc: Kirill A. Shutemov 
Cc: Oscar Salvador 
Cc: Pankaj Gupta 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/sparsemem.c: defer the ms->section_mem_map clearing

2018-08-17T23:20:31+00:00

In sparse_init(), if CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y, system
will allocate one continuous memory chunk for mem maps on one node and
populate the relevant page tables to map memory section one by one.  If
fail to populate for a certain mem section, print warning and its
->section_mem_map will be cleared to cancel the marking of being
present.  Like this, the number of mem sections marked as present could
become less during sparse_init() execution.

Here just defer the ms->section_mem_map clearing if failed to populate
its page tables until the last for_each_present_section_nr() loop.  This
is in preparation for later optimizing the mem map allocation.

[akpm@linux-foundation.org: remove now-unused local `ms', per Oscar]
Link: http://lkml.kernel.org/r/20180228032657.32385-3-bhe@redhat.com
Signed-off-by: Baoquan He 
Acked-by: Dave Hansen 
Reviewed-by: Pavel Tatashin 
Reviewed-by: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Kirill A. Shutemov 
Cc: Pankaj Gupta 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: merge vmem_altmap_alloc into altmap_alloc_block_buf

2018-01-08T19:46:23+00:00

There is no clear separation between the two, so merge them.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Signed-off-by: Dan Williams

mm: split altmap memory map allocation from normal case

2018-01-08T19:46:23+00:00

No functional changes, just untangling the call chain and document
why the altmap is passed around the hotplug code.

Signed-off-by: Christoph Hellwig 
Reviewed-by: Logan Gunthorpe 
Signed-off-by: Dan Williams

mm: pass the vmem_altmap to vmemmap_populate

2018-01-08T19:46:23+00:00

We can just pass this on instead of having to do a radix tree lookup
without proper locking a few levels into the callchain.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Dan Williams

mm, sparse: do not swamp log with huge vmemmap allocation failures

2017-11-16T02:21:07+00:00

While doing memory hotplug tests under heavy memory pressure we have
noticed too many page allocation failures when allocating vmemmap memmap
backed by huge page

  kworker/u3072:1: page allocation failure: order:9, mode:0x24084c0(GFP_KERNEL|__GFP_REPEAT|__GFP_ZERO)
  [...]
  Call Trace:
    dump_trace+0x59/0x310
    show_stack_log_lvl+0xea/0x170
    show_stack+0x21/0x40
    dump_stack+0x5c/0x7c
    warn_alloc_failed+0xe2/0x150
    __alloc_pages_nodemask+0x3ed/0xb20
    alloc_pages_current+0x7f/0x100
    vmemmap_alloc_block+0x79/0xb6
    __vmemmap_alloc_block_buf+0x136/0x145
    vmemmap_populate+0xd2/0x2b9
    sparse_mem_map_populate+0x23/0x30
    sparse_add_one_section+0x68/0x18e
    __add_pages+0x10a/0x1d0
    arch_add_memory+0x4a/0xc0
    add_memory_resource+0x89/0x160
    add_memory+0x6d/0xd0
    acpi_memory_device_add+0x181/0x251
    acpi_bus_attach+0xfd/0x19b
    acpi_bus_scan+0x59/0x69
    acpi_device_hotplug+0xd2/0x41f
    acpi_hotplug_work_fn+0x1a/0x23
    process_one_work+0x14e/0x410
    worker_thread+0x116/0x490
    kthread+0xbd/0xe0
    ret_from_fork+0x3f/0x70

and we do see many of those because essentially every allocation fails
for each memory section.  This is an excessive way to tell the user that
there is nothing to really worry about because we do have a fallback
mechanism to use base pages.  The only downside might be a performance
degradation due to TLB pressure.

This patch changes vmemmap_alloc_block() to use __GFP_NOWARN and warn
explicitly once on the first allocation failure.  This will reduce the
noise in the kernel log considerably, while we still have an indication
that a performance might be impacted.

[mhocko@kernel.org: forgot to git add the follow up fix]
  Link: http://lkml.kernel.org/r/20171107090635.c27thtse2lchjgvb@dhcp22.suse.cz
Link: http://lkml.kernel.org/r/20171106092228.31098-1-mhocko@kernel.org
Signed-off-by: Johannes Weiner 
Signed-off-by: Michal Hocko 
Cc: Joe Perches 
Cc: Vlastimil Babka 
Cc: Khalid Aziz 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: stop zeroing memory during allocation in vmemmap

2017-11-16T02:21:05+00:00

vmemmap_alloc_block() will no longer zero the block, so zero memory at
its call sites for everything except struct pages.  Struct page memory
is zero'd by struct page initialization.

Replace allocators in sparse-vmemmap to use the non-zeroing version.
So, we will get the performance improvement by zeroing the memory in
parallel when struct pages are zeroed.

Add struct page zeroing as a part of initialization of other fields in
__init_single_page().

This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):

                         BASE            FIX
sparse_init     11.244671836s   0.007199623s
zone_sizes_init  4.879775891s   8.355182299s
                  --------------------------
Total           16.124447727s   8.362381922s

sparse_init is where memory for struct pages is zeroed, and the zeroing
part is moved later in this patch into __init_single_page(), which is
called from zone_sizes_init().

[akpm@linux-foundation.org: make vmemmap_alloc_block_zero() private to sparse-vmemmap.c]
Link: http://lkml.kernel.org/r/20171013173214.27300-10-pasha.tatashin@oracle.com
Signed-off-by: Pavel Tatashin 
Reviewed-by: Steven Sistare 
Reviewed-by: Daniel Jordan 
Reviewed-by: Bob Picco 
Tested-by: Bob Picco 
Acked-by: Michal Hocko 
Cc: Alexander Potapenko 
Cc: Andrey Ryabinin 
Cc: Ard Biesheuvel 
Cc: Catalin Marinas 
Cc: Christian Borntraeger 
Cc: David S. Miller 
Cc: Dmitry Vyukov 
Cc: Heiko Carstens 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Mark Rutland 
Cc: Matthew Wilcox 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Sam Ravnborg 
Cc: Thomas Gleixner 
Cc: Will Deacon 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds