linux.git/arch/x86/kernel/cpu/resctrl/internal.h, branch v6.9

x86/resctrl: Fix uninitialized memory read when last CPU of domain goes offline

2024-04-03T07:30:01+00:00

Tony encountered this OOPS when the last CPU of a domain goes
offline while running a kernel built with CONFIG_NO_HZ_FULL:

    BUG: kernel NULL pointer dereference, address: 0000000000000000
    #PF: supervisor read access in kernel mode
    #PF: error_code(0x0000) - not-present page
    PGD 0
    Oops: 0000 [#1] PREEMPT SMP NOPTI
    ...
    RIP: 0010:__find_nth_andnot_bit+0x66/0x110
    ...
    Call Trace:
     
     ? __die()
     ? page_fault_oops()
     ? exc_page_fault()
     ? asm_exc_page_fault()
     cpumask_any_housekeeping()
     mbm_setup_overflow_handler()
     resctrl_offline_cpu()
     resctrl_arch_offline_cpu()
     cpuhp_invoke_callback()
     cpuhp_thread_fun()
     smpboot_thread_fn()
     kthread()
     ret_from_fork()
     ret_from_fork_asm()
     

The NULL pointer dereference is encountered while searching for another
online CPU in the domain (of which there are none) that can be used to
run the MBM overflow handler.

Because the kernel is configured with CONFIG_NO_HZ_FULL the search for
another CPU (in its effort to prefer those CPUs that aren't marked
nohz_full) consults the mask representing the nohz_full CPUs,
tick_nohz_full_mask. On a kernel with CONFIG_CPUMASK_OFFSTACK=y
tick_nohz_full_mask is not allocated unless the kernel is booted with
the "nohz_full=" parameter and because of that any access to
tick_nohz_full_mask needs to be guarded with tick_nohz_full_enabled().

Replace the IS_ENABLED(CONFIG_NO_HZ_FULL) with tick_nohz_full_enabled().
The latter ensures tick_nohz_full_mask can be accessed safely and can be
used whether kernel is built with CONFIG_NO_HZ_FULL enabled or not.

[ Use Ingo's suggestion that combines the two NO_HZ checks into one. ]

Fixes: a4846aaf3945 ("x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflow")
Reported-by: Tony Luck 
Signed-off-by: Reinette Chatre 
Signed-off-by: Ingo Molnar 
Reviewed-by: Babu Moger 
Link: https://lore.kernel.org/r/ff8dfc8d3dcb04b236d523d1e0de13d2ef585223.1711993956.git.reinette.chatre@intel.com
Closes: https://lore.kernel.org/lkml/ZgIFT5gZgIQ9A9G7@agluck-desk3/

x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but CPU

2024-02-16T18:18:33+00:00

When a CPU is taken offline resctrl may need to move the overflow or limbo
handlers to run on a different CPU.

Once the offline callbacks have been split, cqm_setup_limbo_handler() will be
called while the CPU that is going offline is still present in the CPU mask.

Pass the CPU to exclude to cqm_setup_limbo_handler() and
mbm_setup_overflow_handler(). These functions can use a variant of
cpumask_any_but() when selecting the CPU. -1 is used to indicate no CPUs need
excluding.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Babu Moger 
Reviewed-by: Reinette Chatre 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-22-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Add helpers for system wide mon/alloc capable

2024-02-16T18:18:33+00:00

resctrl reads rdt_alloc_capable or rdt_mon_capable to determine whether any of
the resources support the corresponding features.  resctrl also uses the
static keys that affect the architecture's context-switch code to determine the
same thing.

This forces another architecture to have the same static keys.

As the static key is enabled based on the capable flag, and none of the
filesystem uses of these are in the scheduler path, move the capable flags
behind helpers, and use these in the filesystem code instead of the static key.

After this change, only the architecture code manages and uses the static keys
to ensure __resctrl_sched_in() does not need runtime checks.

This avoids multiple architectures having to define the same static keys.

Cases where the static key implicitly tested if the resctrl filesystem was
mounted all have an explicit check now.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-20-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Move alloc/mon static keys into helpers

2024-02-16T18:18:32+00:00

resctrl enables three static keys depending on the features it has enabled.
Another architecture's context switch code may look different, any static keys
that control it should be buried behind helpers.

Move the alloc/mon logic into arch-specific helpers as a preparatory step for
making the rdt_enable_key's status something the arch code decides.

This means other architectures don't have to mirror the static keys.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-18-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Make resctrl_mounted checks explicit

2024-02-16T18:18:32+00:00

The rdt_enable_key is switched when resctrl is mounted, and used to prevent
a second mount of the filesystem. It also enables the architecture's context
switch code.

This requires another architecture to have the same set of static keys, as
resctrl depends on them too. The existing users of these static keys are
implicitly also checking if the filesystem is mounted.

Make the resctrl_mounted checks explicit: resctrl can keep track of whether it
has been mounted once. This doesn't need to be combined with whether the arch
code is context switching the CLOSID.

rdt_mon_enable_key is never used just to test that resctrl is mounted, but does
also have this implication. Add a resctrl_mounted to all uses of
rdt_mon_enable_key.

This will allow the static key changing to be moved behind resctrl_arch_ calls.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-17-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Allow arch to allocate memory needed in resctrl_arch_rmid_read()

2024-02-16T18:18:32+00:00

Depending on the number of monitors available, Arm's MPAM may need to
allocate a monitor prior to reading the counter value. Allocating a
contended resource may involve sleeping.

__check_limbo() and mon_event_count() each make multiple calls to
resctrl_arch_rmid_read(), to avoid extra work on contended systems,
the allocation should be valid for multiple invocations of
resctrl_arch_rmid_read().

The memory or hardware allocated is not specific to a domain.

Add arch hooks for this allocation, which need calling before
resctrl_arch_rmid_read(). The allocated monitor is passed to
resctrl_arch_rmid_read(), then freed again afterwards. The helper
can be called on any CPU, and can sleep.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-16-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflow

2024-02-16T18:18:32+00:00

The limbo and overflow code picks a CPU to use from the domain's list of online
CPUs. Work is then scheduled on these CPUs to maintain the limbo list and any
counters that may overflow.

cpumask_any() may pick a CPU that is marked nohz_full, which will either
penalise the work that CPU was dedicated to, or delay the processing of limbo
list or counters that may overflow. Perhaps indefinitely. Delaying the overflow
handling will skew the bandwidth values calculated by mba_sc, which expects to
be called once a second.

Add cpumask_any_housekeeping() as a replacement for cpumask_any() that prefers
housekeeping CPUs. This helper will still return a nohz_full CPU if that is the
only option. The CPU to use is re-evaluated each time the limbo/overflow work
runs. This ensures the work will move off a nohz_full CPU once a housekeeping
CPU is available.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-13-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Allocate the cleanest CLOSID by searching closid_num_dirty_rmid

2024-02-16T18:18:32+00:00

MPAM's PMG bits extend its PARTID space, meaning the same PMG value can be used
for different control groups.

This means once a CLOSID is allocated, all its monitoring ids may still be
dirty, and held in limbo.

Instead of allocating the first free CLOSID, on architectures where
CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is enabled, search
closid_num_dirty_rmid[] to find the cleanest CLOSID.

The CLOSID found is returned to closid_alloc() for the free list
to be updated.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-11-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Allow RMID allocation to be scoped by CLOSID

2024-02-16T18:18:31+00:00

MPAMs RMID values are not unique unless the CLOSID is considered as well.

alloc_rmid() expects the RMID to be an independent number.

Pass the CLOSID in to alloc_rmid(). Use this to compare indexes when
allocating. If the CLOSID is not relevant to the index, this ends up comparing
the free RMID with itself, and the first free entry will be used. With MPAM the
CLOSID is included in the index, so this becomes a walk of the free RMID
entries, until one that matches the supplied CLOSID is found.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-8-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)

x86/resctrl: Access per-rmid structures by index

2024-02-16T18:18:31+00:00

x86 systems identify traffic using the CLOSID and RMID. The CLOSID is
used to lookup the control policy, the RMID is used for monitoring. For
x86 these are independent numbers.
Arm's MPAM has equivalent features PARTID and PMG, where the PARTID is
used to lookup the control policy. The PMG in contrast is a small number
of bits that are used to subdivide PARTID when monitoring. The
cache-occupancy monitors require the PARTID to be specified when
monitoring.

This means MPAM's PMG field is not unique. There are multiple PMG-0, one
per allocated CLOSID/PARTID. If PMG is treated as equivalent to RMID, it
cannot be allocated as an independent number. Bitmaps like rmid_busy_llc
need to be sized by the number of unique entries for this resource.

Treat the combined CLOSID and RMID as an index, and provide architecture
helpers to pack and unpack an index. This makes the MPAM values unique.
The domain's rmid_busy_llc and rmid_ptrs[] are then sized by index, as
are domain mbm_local[] and mbm_total[].

x86 can ignore the CLOSID field when packing and unpacking an index, and
report as many indexes as RMID.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Babu Moger 
Reviewed-by: Reinette Chatre 
Tested-by: Shaopeng Tan 
Tested-by: Peter Newman 
Tested-by: Babu Moger 
Tested-by: Carl Worth  # arm64
Link: https://lore.kernel.org/r/20240213184438.16675-7-james.morse@arm.com
Signed-off-by: Borislav Petkov (AMD)