linux-stable.git/kernel/sched, branch v4.12.2

sched/numa: Hide numa_wake_affine() from UP build

2017-07-12T14:55:26+00:00

commit ff801b716effd652f420204eddb36f6e4a716819 upstream.

Stephen reported the following build warning in UP:

kernel/sched/fair.c:2657:9: warning: 'struct sched_domain' declared inside
parameter list
         ^
/home/sfr/next/next/kernel/sched/fair.c:2657:9: warning: its scope is only this
definition or declaration, which is probably not what you want

Hide the numa_wake_affine() inline stub on UP builds to get rid of it.

Fixes: 3fed382b46ba ("sched/numa: Implement NUMA node level wake_affine()")
Reported-by: Stephen Rothwell 
Signed-off-by: Thomas Gleixner 
Cc: Rik van Riel 
Cc: Peter Zijlstra 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/fair: Remove effective_load()

2017-07-12T14:55:25+00:00

commit 815abf5af45f04f759f12f3172afd15226fd7f71 upstream.

The effective_load() function was only used by the NUMA balancing
code, and not by the regular load balancing code. Now that the
NUMA balancing code no longer uses it either, get rid of it.

Signed-off-by: Rik van Riel 
Cc: Linus Torvalds 
Cc: Mel Gorman 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: jhladky@redhat.com
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/20170623165530.22514-5-riel@redhat.com
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/numa: Implement NUMA node level wake_affine()

2017-07-12T14:55:25+00:00

commit 3fed382b46baac83703130fe4cd3d9147f427fb9 upstream.

Since select_idle_sibling() can place a task anywhere on a socket,
comparing loads between individual CPU cores makes no real sense
for deciding whether to do an affine wakeup across sockets, either.

Instead, compare the load between the sockets in a similar way the
load balancer and the numa balancing code do.

Signed-off-by: Rik van Riel 
Cc: Linus Torvalds 
Cc: Mel Gorman 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: jhladky@redhat.com
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/20170623165530.22514-4-riel@redhat.com
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/fair: Simplify wake_affine() for the single socket case

2017-07-12T14:55:25+00:00

commit 7d894e6e34a5cdd12309c7e4a3f830277ad4b7bf upstream.

Then 'this_cpu' and 'prev_cpu' are in the same socket, select_idle_sibling()
will do its thing regardless of the return value of wake_affine().

Just return true and don't look at all the other things.

Signed-off-by: Rik van Riel 
Cc: Linus Torvalds 
Cc: Mel Gorman 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: jhladky@redhat.com
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/20170623165530.22514-3-riel@redhat.com
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/numa: Override part of migrate_degrades_locality() when idle balancing

2017-07-12T14:55:25+00:00

commit 739294fb03f590401bbd7faa6d31a507e3ffada5 upstream.

Several tests in the NAS benchmark seem to run a lot slower with
NUMA balancing enabled, than with NUMA balancing disabled. The
slower run time corresponds with increased idle time.

Overriding the final test of migrate_degrades_locality (but still
doing the other NUMA tests first) seems to improve performance
of those benchmarks.

Reported-by: Jirka Hladky 
Signed-off-by: Rik van Riel 
Cc: Linus Torvalds 
Cc: Mel Gorman 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/20170623165530.22514-2-riel@redhat.com
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/numa: Use down_read_trylock() for the mmap_sem

2017-07-12T14:55:25+00:00

commit 8655d5497735b288f8a9b458bd22e7d1bf95bb61 upstream.

A customer has reported a soft-lockup when running an intensive
memory stress test, where the trace on multiple CPU's looks like this:

 RIP: 0010:[]
  [] native_queued_spin_lock_slowpath+0x10e/0x190
...
 Call Trace:
  [] queued_spin_lock_slowpath+0x7/0xa
  [] change_protection_range+0x3b1/0x930
  [] change_prot_numa+0x18/0x30
  [] task_numa_work+0x1fe/0x310
  [] task_work_run+0x72/0x90

Further investigation showed that the lock contention here is pmd_lock().

The task_numa_work() function makes sure that only one thread is let to perform
the work in a single scan period (via cmpxchg), but if there's a thread with
mmap_sem locked for writing for several periods, multiple threads in
task_numa_work() can build up a convoy waiting for mmap_sem for read and then
all get unblocked at once.

This patch changes the down_read() to the trylock version, which prevents the
build up. For a workload experiencing mmap_sem contention, it's probably better
to postpone the NUMA balancing work anyway. This seems to have fixed the soft
lockups involving pmd_lock(), which is in line with the convoy theory.

Signed-off-by: Vlastimil Babka 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Rik van Riel 
Acked-by: Mel Gorman 
Cc: Linus Torvalds 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Link: http://lkml.kernel.org/r/20170515131316.21909-1-vbabka@suse.cz
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/core: Implement new approach to scale select_idle_cpu()

2017-07-12T14:55:25+00:00

commit 1ad3aaf3fcd2444406628a19a9b9e0922b95e2d4 upstream.

Hackbench recently suffered a bunch of pain, first by commit:

  4c77b18cf8b7 ("sched/fair: Make select_idle_cpu() more aggressive")

and then by commit:

  c743f0a5c50f ("sched/fair, cpumask: Export for_each_cpu_wrap()")

which fixed a bug in the initial for_each_cpu_wrap() implementation
that made select_idle_cpu() even more expensive. The bug was that it
would skip over CPUs when bits were consequtive in the bitmask.

This however gave me an idea to fix select_idle_cpu(); where the old
scheme was a cliff-edge throttle on idle scanning, this introduces a
more gradual approach. Instead of stopping to scan entirely, we limit
how many CPUs we scan.

Initial benchmarks show that it mostly recovers hackbench while not
hurting anything else, except Mason's schbench, but not as bad as the
old thing.

It also appears to recover the tbench high-end, which also suffered like
hackbench.

Tested-by: Matt Fleming 
Signed-off-by: Peter Zijlstra (Intel) 
Cc: Chris Mason 
Cc: Linus Torvalds 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: hpa@zytor.com
Cc: kitsunyan 
Cc: linux-kernel@vger.kernel.org
Cc: lvenanci@redhat.com
Cc: riel@redhat.com
Cc: xiaolong.ye@intel.com
Link: http://lkml.kernel.org/r/20170517105350.hk5m4h4jb6dfr65a@hirez.programming.kicks-ass.net
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

sched/fair, cpumask: Export for_each_cpu_wrap()

2017-07-12T14:55:25+00:00

commit c743f0a5c50f2fcbc628526279cfa24f3dabe182 upstream.

More users for for_each_cpu_wrap() have appeared. Promote the construct
to generic cpumask interface.

The implementation is slightly modified to reduce arguments.

Signed-off-by: Peter Zijlstra (Intel) 
Cc: Lauro Ramos Venancio 
Cc: Linus Torvalds 
Cc: Mike Galbraith 
Cc: Peter Zijlstra 
Cc: Rik van Riel 
Cc: Thomas Gleixner 
Cc: lwang@redhat.com
Link: http://lkml.kernel.org/r/20170414122005.o35me2h5nowqkxbv@hirez.programming.kicks-ass.net
Signed-off-by: Ingo Molnar 
Signed-off-by: Mel Gorman 
Signed-off-by: Greg Kroah-Hartman

Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2017-06-18T09:45:17+00:00

Pull scheduler fixes from Thomas Gleixner:
 "Two small fixes for the schedulre core:

   - Use the proper switch_mm() variant in idle_task_exit() because that
     code is not called with interrupts disabled.

   - Fix a confusing typo in a printk"

* 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  sched/core: Idle_task_exit() shouldn't use switch_mm_irqs_off()
  sched/fair: Fix typo in printk message

Merge branches 'pm-cpufreq', 'pm-cpuidle' and 'pm-devfreq'

2017-06-14T23:51:33+00:00

* pm-cpufreq:
  cpufreq: conservative: Allow down_threshold to take values from 1 to 10
  Revert "cpufreq: schedutil: Reduce frequencies slower"

* pm-cpuidle:
  cpuidle: dt: Add missing 'of_node_put()'

* pm-devfreq:
  PM / devfreq: exynos-ppmu: Staticize event list
  PM / devfreq: exynos-ppmu: Handle return value of clk_prepare_enable
  PM / devfreq: exynos-nocp: Handle return value of clk_prepare_enable