linux.git/kernel/sched/deadline.c, branch v5.14

sched/rt: Fix Deadline utilization tracking during policy change

2021-06-22T14:41:59+00:00

DL keeps track of the utilization on a per-rq basis with the structure
avg_dl. This utilization is updated during task_tick_dl(),
put_prev_task_dl() and set_next_task_dl(). However, when the current
running task changes its policy, set_next_task_dl() which would usually
take care of updating the utilization when the rq starts running DL
tasks, will not see a such change, leaving the avg_dl structure outdated.
When that very same task will be dequeued later, put_prev_task_dl() will
then update the utilization, based on a wrong last_update_time, leading to
a huge spike in the DL utilization signal.

The signal would eventually recover from this issue after few ms. Even
if no DL tasks are run, avg_dl is also updated in
__update_blocked_others(). But as the CPU capacity depends partly on the
avg_dl, this issue has nonetheless a significant impact on the scheduler.

Fix this issue by ensuring a load update when a running task changes
its policy to DL.

Fixes: 3727e0e ("sched/dl: Add dl_rq utilization tracking")
Signed-off-by: Vincent Donnefort 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Vincent Guittot 
Link: https://lore.kernel.org/r/1624271872-211872-3-git-send-email-vincent.donnefort@arm.com

sched: Change task_struct::state

2021-06-18T09:43:09+00:00

Change the type and name of task_struct::state. Drop the volatile and
shrink it to an 'unsigned int'. Rename it in order to find all uses
such that we can use READ_ONCE/WRITE_ONCE as appropriate.

Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Daniel Bristot de Oliveira 
Acked-by: Will Deacon 
Acked-by: Daniel Thompson 
Link: https://lore.kernel.org/r/20210611082838.550736351@infradead.org

sched: Introduce sched_class::pick_task()

2021-05-12T09:43:28+00:00

Because sched_class::pick_next_task() also implies
sched_class::set_next_task() (and possibly put_prev_task() and
newidle_balance) it is not state invariant. This makes it unsuitable
for remote task selection.

Signed-off-by: Peter Zijlstra (Intel) 
[Vineeth: folded fixes]
Signed-off-by: Vineeth Remanan Pillai 
Signed-off-by: Peter Zijlstra (Intel) 
Tested-by: Don Hiatt 
Tested-by: Hongyu Ning 
Tested-by: Vincent Guittot 
Link: https://lkml.kernel.org/r/20210422123308.437092775@infradead.org

sched: Optimize rq_lockp() usage

2021-05-12T09:43:27+00:00

rq_lockp() includes a static_branch(), which is asm-goto, which is
asm volatile which defeats regular CSE. This means that:

	if (!static_branch(&foo))
		return simple;

	if (static_branch(&foo) && cond)
		return complex;

Doesn't fold and we get horrible code. Introduce __rq_lockp() without
the static_branch() on.

Signed-off-by: Peter Zijlstra (Intel) 
Tested-by: Don Hiatt 
Tested-by: Hongyu Ning 
Tested-by: Vincent Guittot 
Link: https://lkml.kernel.org/r/20210422123308.316696988@infradead.org

sched: Wrap rq::lock access

2021-05-12T09:43:26+00:00

In preparation of playing games with rq->lock, abstract the thing
using an accessor.

Signed-off-by: Peter Zijlstra (Intel) 
Tested-by: Don Hiatt 
Tested-by: Hongyu Ning 
Tested-by: Vincent Guittot 
Link: https://lkml.kernel.org/r/20210422123308.136465446@infradead.org

sched: Fix various typos

2021-03-21T23:11:52+00:00

Fix ~42 single-word typos in scheduler code comments.

We have accumulated a few fun ones over the years. :-)

Signed-off-by: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Mike Galbraith 
Cc: Juri Lelli 
Cc: Vincent Guittot 
Cc: Dietmar Eggemann 
Cc: Steven Rostedt 
Cc: Ben Segall 
Cc: Mel Gorman 
Cc: linux-kernel@vger.kernel.org

sched/features: Distinguish between NORMAL and DEADLINE hrtick

2021-02-17T13:12:42+00:00

The HRTICK feature has traditionally been servicing configurations that
need precise preemptions point for NORMAL tasks. More recently, the
feature has been extended to also service DEADLINE tasks with stringent
runtime enforcement needs (e.g., runtime < 1ms with HZ=1000).

Enabling HRTICK sched feature currently enables the additional timer and
task tick for both classes, which might introduced undesired overhead
for no additional benefit if one needed it only for one of the cases.

Separate HRTICK sched feature in two (and leave the traditional case
name unmodified) so that it can be selectively enabled when needed.

With:

  $ echo HRTICK > /sys/kernel/debug/sched_features

the NORMAL/fair hrtick gets enabled.

With:

  $ echo HRTICK_DL > /sys/kernel/debug/sched_features

the DEADLINE hrtick gets enabled.

Signed-off-by: Juri Lelli 
Signed-off-by: Luis Claudio R. Goncalves 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Link: https://lkml.kernel.org/r/20210208073554.14629-3-juri.lelli@redhat.com

sched/deadline: Reduce rq lock contention in dl_add_task_root_domain()

2021-02-17T13:12:42+00:00

dl_add_task_root_domain() is called during sched domain rebuild:

  rebuild_sched_domains_locked()
    partition_and_rebuild_sched_domains()
      rebuild_root_domains()
         for all top_cpuset descendants:
           update_tasks_root_domain()
             for all tasks of cpuset:
               dl_add_task_root_domain()

Change it so that only the task pi lock is taken to check if the task
has a SCHED_DEADLINE (DL) policy. In case that p is a DL task take the
rq lock as well to be able to safely de-reference root domain's DL
bandwidth structure.

Most of the tasks will have another policy (namely SCHED_NORMAL) and
can now bail without taking the rq lock.

One thing to note here: Even in case that there aren't any DL user
tasks, a slow frequency switching system with cpufreq gov schedutil has
a DL task (sugov) per frequency domain running which participates in DL
bandwidth management.

Signed-off-by: Dietmar Eggemann 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Quentin Perret 
Reviewed-by: Valentin Schneider 
Reviewed-by: Daniel Bristot de Oliveira 
Acked-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20210119083542.19856-1-dietmar.eggemann@arm.com

rbtree, sched/deadline: Use rb_add_cached()

2021-02-17T13:07:44+00:00

Reduce rbtree boiler plate by using the new helpers.

Make rb_add_cached() / rb_erase_cached() return a pointer to the
leftmost node to aid in updating additional state.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Acked-by: Davidlohr Bueso

sched: Use task_current() instead of 'rq->curr == p'

2021-01-14T10:20:11+00:00

Use the task_current() function where appropriate.

No functional change.

Signed-off-by: Hui Su 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Steven Rostedt (VMware) 
Link: https://lkml.kernel.org/r/20201030173223.GA52339@rlk