linux.git/arch/x86/kernel/cpu/resctrl/rdtgroup.c, branch v6.2

x86/resctrl: Fix task CLOSID/RMID update race

2023-01-10T18:47:30+00:00

When the user moves a running task to a new rdtgroup using the task's
file interface or by deleting its rdtgroup, the resulting change in
CLOSID/RMID must be immediately propagated to the PQR_ASSOC MSR on the
task(s) CPUs.

x86 allows reordering loads with prior stores, so if the task starts
running between a task_curr() check that the CPU hoisted before the
stores in the CLOSID/RMID update then it can start running with the old
CLOSID/RMID until it is switched again because __rdtgroup_move_task()
failed to determine that it needs to be interrupted to obtain the new
CLOSID/RMID.

Refer to the diagram below:

CPU 0                                   CPU 1
-----                                   -----
__rdtgroup_move_task():
  curr <- t1->cpu->rq->curr
                                        __schedule():
                                          rq->curr <- t1
                                        resctrl_sched_in():
                                          t1->{closid,rmid} -> {1,1}
  t1->{closid,rmid} <- {2,2}
  if (curr == t1) // false
   IPI(t1->cpu)

A similar race impacts rdt_move_group_tasks(), which updates tasks in a
deleted rdtgroup.

In both cases, use smp_mb() to order the task_struct::{closid,rmid}
stores before the loads in task_curr().  In particular, in the
rdt_move_group_tasks() case, simply execute an smp_mb() on every
iteration with a matching task.

It is possible to use a single smp_mb() in rdt_move_group_tasks(), but
this would require two passes and a means of remembering which
task_structs were updated in the first loop. However, benchmarking
results below showed too little performance impact in the simple
approach to justify implementing the two-pass approach.

Times below were collected using `perf stat` to measure the time to
remove a group containing a 1600-task, parallel workload.

CPU: Intel(R) Xeon(R) Platinum P-8136 CPU @ 2.00GHz (112 threads)

  # mkdir /sys/fs/resctrl/test
  # echo $$ > /sys/fs/resctrl/test/tasks
  # perf bench sched messaging -g 40 -l 100000

task-clock time ranges collected using:

  # perf stat rmdir /sys/fs/resctrl/test

Baseline:                     1.54 - 1.60 ms
smp_mb() every matching task: 1.57 - 1.67 ms

  [ bp: Massage commit message. ]

Fixes: ae28d1aae48a ("x86/resctrl: Use an IPI instead of task_work_add() to update PQR_ASSOC MSR")
Fixes: 0efc89be9471 ("x86/intel_rdt: Update task closid immediately on CPU in rmdir and unmount")
Signed-off-by: Peter Newman 
Signed-off-by: Borislav Petkov (AMD) 
Reviewed-by: Reinette Chatre 
Reviewed-by: Babu Moger 
Cc: 
Link: https://lore.kernel.org/r/20221220161123.432120-1-peternewman@google.com

x86/resctrl: Add resctrl_rmid_realloc_limit to abstract x86's boot_cpu_data

2022-09-23T12:24:16+00:00

resctrl_rmid_realloc_threshold can be set by user-space. The maximum
value is specified by the architecture.

Currently max_threshold_occ_write() reads the maximum value from
boot_cpu_data.x86_cache_size, which is not portable to another
architecture.

Add resctrl_rmid_realloc_limit to describe the maximum size in bytes
that user-space can set the threshold to.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-21-james.morse@arm.com

x86/resctrl: Rename and change the units of resctrl_cqm_threshold

2022-09-23T12:23:41+00:00

resctrl_cqm_threshold is stored in a hardware specific chunk size,
but exposed to user-space as bytes.

This means the filesystem parts of resctrl need to know how the hardware
counts, to convert the user provided byte value to chunks. The interface
between the architecture's resctrl code and the filesystem ought to
treat everything as bytes.

Change the unit of resctrl_cqm_threshold to bytes. resctrl_arch_rmid_read()
still returns its value in chunks, so this needs converting to bytes.
As all the users have been touched, rename the variable to
resctrl_rmid_realloc_threshold, which describes what the value is for.

Neither r->num_rmid nor hw_res->mon_scale are guaranteed to be a power
of 2, so the existing code introduces a rounding error from resctrl's
theoretical fraction of the cache usage. This behaviour is kept as it
ensures the user visible value matches the value read from hardware
when the rmid will be reallocated.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-20-james.morse@arm.com

x86/resctrl: Remove architecture copy of mbps_val

2022-09-22T15:37:16+00:00

The resctrl arch code provides a second configuration array mbps_val[]
for the MBA software controller.

Since resctrl switched over to allocating and freeing its own array
when needed, nothing uses the arch code version.

Remove it.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-11-james.morse@arm.com

x86/resctrl: Switch over to the resctrl mbps_val list

2022-09-22T15:34:08+00:00

Updates to resctrl's software controller follow the same path as
other configuration updates, but they don't modify the hardware state.
rdtgroup_schemata_write() uses parse_line() and the resource's
parse_ctrlval() function to stage the configuration.
resctrl_arch_update_domains() then updates the mbps_val[] array
instead, and resctrl_arch_update_domains() skips the rdt_ctrl_update()
call that would update hardware.

This complicates the interface between resctrl's filesystem parts
and architecture specific code. It should be possible for mba_sc
to be completely implemented by the filesystem parts of resctrl. This
would allow it to work on a second architecture with no additional code.
resctrl_arch_update_domains() using the mbps_val[] array prevents this.

Change parse_bw() to write the configuration value directly to the
mbps_val[] array in the domain structure. Change rdtgroup_schemata_write()
to skip the call to resctrl_arch_update_domains(), meaning all the
mba_sc specific code in resctrl_arch_update_domains() can be removed.
On the read-side, show_doms() and update_mba_bw() are changed to read
the mbps_val[] array from the domain structure. With this,
resctrl_arch_get_config() no longer needs to consider mba_sc resources.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-10-james.morse@arm.com

x86/resctrl: Create mba_sc configuration in the rdt_domain

2022-09-22T15:17:59+00:00

To support resctrl's MBA software controller, the architecture must provide
a second configuration array to hold the mbps_val[] from user-space.

This complicates the interface between the architecture specific code and
the filesystem portions of resctrl that will move to /fs/, to allow
multiple architectures to support resctrl.

Make the filesystem parts of resctrl create an array for the mba_sc
values. The software controller can be changed to use this, allowing
the architecture code to only consider the values configured in hardware.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-9-james.morse@arm.com

x86/resctrl: Abstract and use supports_mba_mbps()

2022-09-22T14:10:11+00:00

To determine whether the mba_MBps option to resctrl should be supported,
resctrl tests the boot CPUs' x86_vendor.

This isn't portable, and needs abstracting behind a helper so this check
can be part of the filesystem code that moves to /fs/.

Re-use the tests set_mba_sc() does to determine if the mba_sc is supported
on this system. An 'alloc_capable' test is added so that support for the
controls isn't implied by the 'delay_linear' property, which is always
true for MPAM. Because mbm_update() only update mba_sc if the mbm_local
counters are enabled, supports_mba_mbps() checks is_mbm_local_enabled().
(instead of using is_mbm_enabled(), which checks both).

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-8-james.morse@arm.com

x86/resctrl: Remove set_mba_sc()s control array re-initialisation

2022-09-22T14:08:20+00:00

set_mba_sc() enables the 'software controller' to regulate the bandwidth
based on the byte counters. This can be managed entirely in the parts
of resctrl that move to /fs/, without any extra support from the
architecture specific code. set_mba_sc() is called by rdt_enable_ctx()
during mount and unmount. It currently resets the arch code's ctrl_val[]
and mbps_val[] arrays.

The ctrl_val[] was already reset when the domain was created, and by
reset_all_ctrls() when the filesystem was last unmounted. Doing the work
in set_mba_sc() is not necessary as the values are already at their
defaults due to the creation of the domain, or were previously reset
during umount(), or are about to reset during umount().

Add a reset of the mbps_val[] in reset_all_ctrls(), allowing the code in
set_mba_sc() that reaches in to the architecture specific structures to
be removed.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-7-james.morse@arm.com

x86/resctrl: Add domain offline callback for resctrl work

2022-09-22T13:42:40+00:00

Because domains are exposed to user-space via resctrl, the filesystem
must update its state when CPU hotplug callbacks are triggered.

Some of this work is common to any architecture that would support
resctrl, but the work is tied up with the architecture code to
free the memory.

Move the monitor subdir removal and the cancelling of the mbm/limbo
works into a new resctrl_offline_domain() call. These bits are not
specific to the architecture. Grouping them in one function allows
that code to be moved to /fs/ and re-used by another architecture.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-6-james.morse@arm.com

x86/resctrl: Add domain online callback for resctrl work

2022-09-22T13:13:27+00:00

Because domains are exposed to user-space via resctrl, the filesystem
must update its state when CPU hotplug callbacks are triggered.

Some of this work is common to any architecture that would support
resctrl, but the work is tied up with the architecture code to
allocate the memory.

Move domain_setup_mon_state(), the monitor subdir creation call and the
mbm/limbo workers into a new resctrl_online_domain() call. These bits
are not specific to the architecture. Grouping them in one function
allows that code to be moved to /fs/ and re-used by another architecture.

Signed-off-by: James Morse 
Signed-off-by: Borislav Petkov 
Reviewed-by: Jamie Iles 
Reviewed-by: Shaopeng Tan 
Reviewed-by: Reinette Chatre 
Tested-by: Xin Hao 
Tested-by: Shaopeng Tan 
Tested-by: Cristian Marussi 
Link: https://lore.kernel.org/r/20220902154829.30399-4-james.morse@arm.com