linux-stable.git/mm/vmstat.c, branch v3.16.65

mm/vmstat.c: skip NR_TLB_REMOTE_FLUSH* properly

2018-12-16T22:09:37+00:00

commit 58bc4c34d249bf1bc50730a9a209139347cfacfe upstream.

5dd0b16cdaff ("mm/vmstat: Make NR_TLB_REMOTE_FLUSH_RECEIVED available even
on UP") made the availability of the NR_TLB_REMOTE_FLUSH* counters inside
the kernel unconditional to reduce #ifdef soup, but (either to avoid
showing dummy zero counters to userspace, or because that code was missed)
didn't update the vmstat_array, meaning that all following counters would
be shown with incorrect values.

This only affects kernel builds with
CONFIG_VM_EVENT_COUNTERS=y && CONFIG_DEBUG_TLBFLUSH=y && CONFIG_SMP=n.

Link: http://lkml.kernel.org/r/20181001143138.95119-2-jannh@google.com
Fixes: 5dd0b16cdaff ("mm/vmstat: Make NR_TLB_REMOTE_FLUSH_RECEIVED available even on UP")
Signed-off-by: Jann Horn 
Reviewed-by: Kees Cook 
Reviewed-by: Andrew Morton 
Acked-by: Michal Hocko 
Acked-by: Roman Gushchin 
Cc: Davidlohr Bueso 
Cc: Oleg Nesterov 
Cc: Christoph Lameter 
Cc: Kemi Wang 
Cc: Andy Lutomirski 
Cc: Ingo Molnar 
Signed-off-by: Andrew Morton 
Signed-off-by: Greg Kroah-Hartman 
Signed-off-by: Ben Hutchings

KAISER: Kernel Address Isolation

2018-01-09T00:35:13+00:00

This patch introduces our implementation of KAISER (Kernel Address Isolation to
have Side-channels Efficiently Removed), a kernel isolation technique to close
hardware side channels on kernel address information.

More information about the patch can be found on:

        https://github.com/IAIK/KAISER

From: Richard Fellner 
From: Daniel Gruss 
Subject: [RFC, PATCH] x86_64: KAISER - do not map kernel in user mode
Date: Thu, 4 May 2017 14:26:50 +0200
Link: http://marc.info/?l=linux-kernel&m=149390087310405&w=2
Kaiser-4.10-SHA1: c4b1831d44c6144d3762ccc72f0c4e71a0c713e5

To: 
To: 
Cc: 
Cc: 
Cc: Michael Schwarz 
Cc: Richard Fellner 
Cc: Ingo Molnar 
Cc: 
Cc: 

After several recent works [1,2,3] KASLR on x86_64 was basically
considered dead by many researchers. We have been working on an
efficient but effective fix for this problem and found that not mapping
the kernel space when running in user mode is the solution to this
problem [4] (the corresponding paper [5] will be presented at ESSoS17).

With this RFC patch we allow anybody to configure their kernel with the
flag CONFIG_KAISER to add our defense mechanism.

If there are any questions we would love to answer them.
We also appreciate any comments!

Cheers,
Daniel (+ the KAISER team from Graz University of Technology)

[1] http://www.ieee-security.org/TC/SP2013/papers/4977a191.pdf
[2] https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-Using-Undocumented-CPU-Behaviour-To-See-Into-Kernel-Mode-And-Break-KASLR-In-The-Process.pdf
[3] https://www.blackhat.com/docs/us-16/materials/us-16-Jang-Breaking-Kernel-Address-Space-Layout-Randomization-KASLR-With-Intel-TSX.pdf
[4] https://github.com/IAIK/KAISER
[5] https://gruss.cc/files/kaiser.pdf

(cherry picked from Change-Id: I0eb000c33290af01fc4454ca0c701d00f1d30b1d)

Conflicts:
arch/x86/entry/entry_64.S (not in this tree)
arch/x86/kernel/entry_64.S (patched instead of that)
arch/x86/entry/entry_64_compat.S (not in this tree)
arch/x86/ia32/ia32entry.S (patched instead of that)
arch/x86/include/asm/hw_irq.h
arch/x86/include/asm/pgtable_types.h
arch/x86/include/asm/processor.h
arch/x86/kernel/irqinit.c
arch/x86/kernel/process.c
arch/x86/mm/Makefile
arch/x86/mm/pgtable.c
init/main.c

Signed-off-by: Hugh Dickins 
[bwh: Folded in the follow-up patches from Hugh:
 - kaiser: merged update
 - kaiser: do not set _PAGE_NX on pgd_none
 - kaiser: stack map PAGE_SIZE at THREAD_SIZE-PAGE_SIZE
 - kaiser: fix build and FIXME in alloc_ldt_struct()
 - kaiser: KAISER depends on SMP
 - kaiser: fix regs to do_nmi() ifndef CONFIG_KAISER
 - kaiser: fix perf crashes
 - kaiser: ENOMEM if kaiser_pagetable_walk() NULL
 - kaiser: tidied up asm/kaiser.h somewhat
 - kaiser: tidied up kaiser_add/remove_mapping slightly
 - kaiser: kaiser_remove_mapping() move along the pgd
 - kaiser: align addition to x86/mm/Makefile
 - kaiser: cleanups while trying for gold link
 - kaiser: name that 0x1000 KAISER_SHADOW_PGD_OFFSET
 - kaiser: delete KAISER_REAL_SWITCH option
 - kaiser: vmstat show NR_KAISERTABLE as nr_overhead
 - kaiser: enhanced by kernel and user PCIDs
 - kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user
 - kaiser: PCID 0 for kernel and 128 for user
 - kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user
 - kaiser: paranoid_entry pass cr3 need to paranoid_exit
 - kaiser: _pgd_alloc() without __GFP_REPEAT to avoid stalls
 - kaiser: fix unlikely error in alloc_ldt_struct()
 - kaiser: drop is_atomic arg to kaiser_pagetable_walk()
 Backported to 3.16:
 - Add missing #include in arch/x86/mm/kaiser.c
 - Use variable PEBS buffer size since we have "perf/x86/intel: Use PAGE_SIZE
   for PEBS buffer size on Core2"
 - Renumber X86_FEATURE_INVPCID_SINGLE to avoid collision
 - Adjust context]
Signed-off-by: Ben Hutchings

vmstat: explicitly schedule per-cpu work on the CPU we need it to run on

2016-02-24T10:26:37+00:00

commit 176bed1de5bf977938cad26551969eca8f0883b1 upstream.

The vmstat code uses "schedule_delayed_work_on()" to do the initial
startup of the delayed work on the right CPU, but then once it was
started it would use the non-cpu-specific "schedule_delayed_work()" to
re-schedule it on that CPU.

That just happened to schedule it on the same CPU historically (well, in
almost all situations), but the code _requires_ this work to be per-cpu,
and should say so explicitly rather than depend on the non-cpu-specific
scheduling to schedule on the current CPU.

The timer code is being changed to not be as single-minded in always
running things on the calling CPU.

See also commit 874bbfe600a6 ("workqueue: make sure delayed work run in
local cpu") that for now maintains the local CPU guarantees just in case
there are other broken users that depended on the accidental behavior.

js: 3.12 backport

Cc: Christoph Lameter 
Cc: Tejun Heo 
Cc: Mike Galbraith 
Cc: Michal Hocko 
Signed-off-by: Linus Torvalds 
Signed-off-by: Jiri Slaby 
[ kamal: backport to 3.16-stable: use queue_delayed_work_on() ]
Signed-off-by: Kamal Mostafa 
Signed-off-by: Luis Henriques

vmstat: allocate vmstat_wq before it is used

2016-01-25T10:43:53+00:00

commit 751e5f5c753e8d447bcf89f9e96b9616ac081628 upstream.

kernel test robot has reported the following crash:

  BUG: unable to handle kernel NULL pointer dereference at 00000100
  IP: [] __queue_work+0x26/0x390
  *pdpt = 0000000000000000 *pde = f000ff53f000ff53 *pde = f000ff53f000ff53
  Oops: 0000 [#1] PREEMPT PREEMPT SMP SMP
  CPU: 0 PID: 24 Comm: kworker/0:1 Not tainted 4.4.0-rc4-00139-g373ccbe #1
  Workqueue: events vmstat_shepherd
  task: cb684600 ti: cb7ba000 task.ti: cb7ba000
  EIP: 0060:[] EFLAGS: 00010046 CPU: 0
  EIP is at __queue_work+0x26/0x390
  EAX: 00000046 EBX: cbb37800 ECX: cbb37800 EDX: 00000000
  ESI: 00000000 EDI: 00000000 EBP: cb7bbe68 ESP: cb7bbe38
   DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
  CR0: 8005003b CR2: 00000100 CR3: 01fd5000 CR4: 000006b0
  Stack:
  Call Trace:
    __queue_delayed_work+0xa1/0x160
    queue_delayed_work_on+0x36/0x60
    vmstat_shepherd+0xad/0xf0
    process_one_work+0x1aa/0x4c0
    worker_thread+0x41/0x440
    kthread+0xb0/0xd0
    ret_from_kernel_thread+0x21/0x40

The reason is that start_shepherd_timer schedules the shepherd work item
which uses vmstat_wq (vmstat_shepherd) before setup_vmstat allocates
that workqueue so if the further initialization takes more than HZ we
might end up scheduling on a NULL vmstat_wq.  This is really unlikely
but not impossible.

Fixes: 373ccbe59270 ("mm, vmstat: allow WQ concurrency to discover memory reclaim doesn't make any progress")
Reported-by: kernel test robot 
Signed-off-by: Michal Hocko 
Tested-by: Tetsuo Handa 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds 
Cc: Ben Hutchings 
[ luis: backported to 3.16: based on Ben's backport to 3.2:
  - as with 3.2, there's a similar race but with the CPU hotplug code ]
Signed-off-by: Luis Henriques

mm, vmstat: allow WQ concurrency to discover memory reclaim doesn't make any progress

2016-01-25T10:42:14+00:00

commit 373ccbe5927034b55bdc80b0f8b54d6e13fe8d12 upstream.

Tetsuo Handa has reported that the system might basically livelock in
OOM condition without triggering the OOM killer.

The issue is caused by internal dependency of the direct reclaim on
vmstat counter updates (via zone_reclaimable) which are performed from
the workqueue context.  If all the current workers get assigned to an
allocation request, though, they will be looping inside the allocator
trying to reclaim memory but zone_reclaimable can see stalled numbers so
it will consider a zone reclaimable even though it has been scanned way
too much.  WQ concurrency logic will not consider this situation as a
congested workqueue because it relies that worker would have to sleep in
such a situation.  This also means that it doesn't try to spawn new
workers or invoke the rescuer thread if the one is assigned to the
queue.

In order to fix this issue we need to do two things.  First we have to
let wq concurrency code know that we are in trouble so we have to do a
short sleep.  In order to prevent from issues handled by 0e093d99763e
("writeback: do not sleep on the congestion queue if there are no
congested BDIs or if significant congestion is not being encountered in
the current zone") we limit the sleep only to worker threads which are
the ones of the interest anyway.

The second thing to do is to create a dedicated workqueue for vmstat and
mark it WQ_MEM_RECLAIM to note it participates in the reclaim and to
have a spare worker thread for it.

Signed-off-by: Michal Hocko 
Reported-by: Tetsuo Handa 
Cc: Tejun Heo 
Cc: Cristopher Lameter 
Cc: Joonsoo Kim 
Cc: Arkadiusz Miskiewicz 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds 
Cc: Ben Hutchings 
[ luis: backported to 3.16, based on Ben's backport to 3.2:
  - use queue_delayed_work instead of queue_delayed_work_on in function
    vmstat_update()
  - change start_cpu_timer() instead of vmstat_shepherd()
  - adjusted context ]
Signed-off-by: Luis Henriques

mm: use the light version __mod_zone_page_state in mlocked_vma_newpage()

2014-06-04T23:54:07+00:00

mlocked_vma_newpage() is called with pte lock held(a spinlock), which
implies preemtion disabled, and the vm stat counter is not modified from
interrupt context, so we need not use an irq-safe mod_zone_page_state()
here, using a light-weight version __mod_zone_page_state() would be OK.

This patch also documents __mod_zone_page_state() and some of its
callsites.  The comment above __mod_zone_page_state() is from Hugh
Dickins, and acked by Christoph.

Most credits to Hugh and Christoph for the clarification on the usage of
the __mod_zone_page_state().

[akpm@linux-foundation.org: coding-style fixes]
Suggested-by: Andrew Morton 
Acked-by: Hugh Dickins 
Signed-off-by: Jianyu Zhan 
Reviewed-by: Christoph Lameter 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: replace __get_cpu_var uses with this_cpu_ptr

2014-06-04T23:54:03+00:00

Replace places where __get_cpu_var() is used for an address calculation
with this_cpu_ptr().

Signed-off-by: Christoph Lameter 
Cc: Tejun Heo 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm,vmacache: add debug data

2014-06-04T23:53:57+00:00

Introduce a CONFIG_DEBUG_VM_VMACACHE option to enable counting the cache
hit rate -- exported in /proc/vmstat.

Any updates to the caching scheme needs this kind of data, thus it can
save some work re-implementing the counting all the time.

Signed-off-by: Davidlohr Bueso 
Cc: Aswin Chandramouleeswaran 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

Merge tag 'cpu-hotplug-3.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm

2014-04-07T21:55:46+00:00

Pull CPU hotplug notifiers registration fixes from Rafael Wysocki:
 "The purpose of this single series of commits from Srivatsa S Bhat
  (with a small piece from Gautham R Shenoy) touching multiple
  subsystems that use CPU hotplug notifiers is to provide a way to
  register them that will not lead to deadlocks with CPU online/offline
  operations as described in the changelog of commit 93ae4f978ca7f ("CPU
  hotplug: Provide lockless versions of callback registration
  functions").

  The first three commits in the series introduce the API and document
  it and the rest simply goes through the users of CPU hotplug notifiers
  and converts them to using the new method"

* tag 'cpu-hotplug-3.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (52 commits)
  net/iucv/iucv.c: Fix CPU hotplug callback registration
  net/core/flow.c: Fix CPU hotplug callback registration
  mm, zswap: Fix CPU hotplug callback registration
  mm, vmstat: Fix CPU hotplug callback registration
  profile: Fix CPU hotplug callback registration
  trace, ring-buffer: Fix CPU hotplug callback registration
  xen, balloon: Fix CPU hotplug callback registration
  hwmon, via-cputemp: Fix CPU hotplug callback registration
  hwmon, coretemp: Fix CPU hotplug callback registration
  thermal, x86-pkg-temp: Fix CPU hotplug callback registration
  octeon, watchdog: Fix CPU hotplug callback registration
  oprofile, nmi-timer: Fix CPU hotplug callback registration
  intel-idle: Fix CPU hotplug callback registration
  clocksource, dummy-timer: Fix CPU hotplug callback registration
  drivers/base/topology.c: Fix CPU hotplug callback registration
  acpi-cpufreq: Fix CPU hotplug callback registration
  zsmalloc: Fix CPU hotplug callback registration
  scsi, fcoe: Fix CPU hotplug callback registration
  scsi, bnx2fc: Fix CPU hotplug callback registration
  scsi, bnx2i: Fix CPU hotplug callback registration
  ...

drop_caches: add some documentation and info message

2014-04-03T23:21:04+00:00

There is plenty of anecdotal evidence and a load of blog posts
suggesting that using "drop_caches" periodically keeps your system
running in "tip top shape".  Perhaps adding some kernel documentation
will increase the amount of accurate data on its use.

If we are not shrinking caches effectively, then we have real bugs.
Using drop_caches will simply mask the bugs and make them harder to
find, but certainly does not fix them, nor is it an appropriate
"workaround" to limit the size of the caches.  On the contrary, there
have been bug reports on issues that turned out to be misguided use of
cache dropping.

Dropping caches is a very drastic and disruptive operation that is good
for debugging and running tests, but if it creates bug reports from
production use, kernel developers should be aware of its use.

Add a bit more documentation about it, a syslog message to track down
abusers, and vmstat drop counters to help analyze problem reports.

[akpm@linux-foundation.org: checkpatch fixes]
[hannes@cmpxchg.org: add runtime suppression control]
Signed-off-by: Dave Hansen 
Signed-off-by: Michal Hocko 
Acked-by: KOSAKI Motohiro 
Acked-by: KAMEZAWA Hiroyuki 
Signed-off-by: Johannes Weiner 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds