linux-stable.git/tools/tracing, branch v6.5.2

rtla/timerlat_hist: Add timerlat user-space support

2023-06-13T20:41:14+00:00

Add the support for running timerlat threads in user-space. In this
mode, enabled with -u/--user-threads, timerlat dispatches user-space
processes that will loop in the timerlat_fd, measuring the overhead
for going to user-space and then returning to the kernel - in addition
to the existing measurements.

Here is one example of the tool's output with -u enabled:

  $ sudo timerlat  hist -u -c 1-3 -d 600
  # RTLA timerlat histogram
  # Time unit is microseconds (us)
  # Duration:   0 00:10:01
  Index   IRQ-001   Thr-001   Usr-001   IRQ-002   Thr-002   Usr-002   IRQ-003   Thr-003   Usr-003
  0        477555         0         0    425287         0         0    474357         0         0
  1        122385      7998         0    174616      1921         0    125412      3138         0
  2            47    587376    492150        89    594717    447830       147    593463    454872
  3            11      2549    101930         7      2682    145580        64      2530    138680
  4             3      1954      2833         1       463      4917        11       548      4656
  5             0        60      1037         0       138      1117         6       179      1130
  6             0        26      1837         0        38       277         1        76       339
  7             0        15       143         0        28       147         2        37       156
  8             0        10        23         0        11        75         0        12        80
  9             0         7        17         0         0        26         0        11        42
  10            0         2        11         0         0        18         0         2        20
  11            0         0         7         0         1         8         0         2        12
  12            0         0         6         0         1         4         0         2         8
  13            0         1         3         0         0         0         0         0         1
  14            0         1         0         0         0         1         0         0         2
  15            0         1         0         0         0         0         0         0         2
  16            0         1         2         0         0         0         0         0         0
  17            0         0         1         0         0         0         0         0         0
  19            0         0         1         0         0         0         0         0         0
  over:         0         0         0         0         0         0         0         0         0
  count:   600001    600001    600001    600000    600000    600000    600000    600000    600000
  min:          0         1         2         0         1         2         0         1         2
  avg:          0         1         2         0         2         2         0         2         2
  max:          4        16        19         4        12        14         7        12        15

The tuning setup like -p or -C work for the user-space threads as well.

Link: https://lkml.kernel.org/r/b6a042d55003c4a67ff7dce28d96044b7044f00d.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat_top: Add timerlat user-space support

2023-06-13T20:38:51+00:00

Add the support for running timerlat threads in user-space. In this
mode, enabled with -u/--user-threads, timerlat dispatches user-space
processes that will loop in the timerlat_fd, measuring the overhead
for going to user-space and then returning to the kernel - in addition
to the existing measurements.

Here is one example of the tool's output with -u enabled:

  $ sudo timerlat top -u -d 600 -q
                                       Timer Latency
    0 00:10:01   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)      |    Ret user Timer Latency (us)
  CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max |      cur       min       avg       max
    0 #600001    |        0         0         0         3 |        2         1         2         9 |        3         2         3        15
    1 #600001    |        0         0         0         2 |        2         1         2        13 |        2         2         3        18
    2 #600001    |        0         0         0        10 |        2         1         2        16 |        3         2         3        20
    3 #600001    |        0         0         0         7 |        2         1         2        10 |        3         2         3        11
    4 #600000    |        0         0         0        16 |        2         1         2        41 |        3         2         3        58
    5 #600000    |        0         0         0         3 |        2         1         2        10 |        3         2         3        13
    6 #600000    |        0         0         0         5 |        2         1         2         7 |        3         2         3        10
    7 #600000    |        0         0         0         1 |        2         1         2         7 |        3         2         3        10

The tuning setup like -p or -C work for the user-space threads as well.

Link: https://lkml.kernel.org/r/758ad2292a0a1d884138d08219e1a0f572d257a2.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla/hwnoise: Reduce runtime to 75%

2023-06-13T20:38:11+00:00

osnoise runs 100% of time by default. It makes sense because osnoise
is preemptive. hwnoise checks preemption once a second, so it
reduces system progress.

Reduce runtime to 75% to avoid problems by default. I added a Fixes
as it might avoid problems for first time users as it lands on distros.

Link: https://lkml.kernel.org/r/af0b7113ffc00031b9af4bb40ef5889a27dadf8c.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Fixes: 1f428356c38d ("rtla: Add hwnoise tool")
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla: Start the tracers after creating all instances

2023-06-13T20:37:05+00:00

Group all start tracing after finishing creating all instances.

The tracing instance starts first for the case of hitting a stop
tracing while enabling other instances. The trace instance is the
one with most valuable information.

Link: https://lkml.kernel.org/r/67da7a703a56f75d7cd46568525145a65501a7e8.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat_hist: Add auto-analysis support

2023-06-13T20:35:46+00:00

Add auto-analysis to timerlat hist, including the --no-aa option to
reduce overhead and --dump-task. --aa-only was not added as it is
already on timerlat top.

Link: https://lkml.kernel.org/r/c2693f47ee83e659a7723fed8035f5d2534f528e.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Give timerlat auto analysis its own instance

2023-06-13T20:31:35+00:00

Currently, the auto-analysis is attached to the timerlat top instance.
The idea was to avoid creating another instance just for that, so one
instance could be reused.

The drawback is that, by doing so, the auto-analysis run for the entire
session, consuming CPU time. On my 24 box CPUs for timerlat with a 100
us period consumed 50 % with auto analysis, but only 16 % without.

By creating an instance for auto-analysis, we can keep the processing
stopped until a stop tracing condition is hit. Once it happens,
timerlat auto-analysis can use its own trace instance to parse only
the end of the trace.

By doing so, auto-analysis stop consuming cpu time when it is not
needed.

If the --aa-only is passed, the timerlat top instance is reused for
auto analysis.

Link: https://lkml.kernel.org/r/346b7168c1bae552a415715ec6d23c129a43bdb7.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla: Automatically move rtla to a house-keeping cpu

2023-06-13T20:30:13+00:00

When the user sets -c  try to move rtla out of the ,
even without an -H option. This is useful to avoid having rtla
interfering with the workload.

This works by removing  from rtla's current affinity.

If rtla fails to move itself away it is not that of a problem as this
is an automatic measure.

Link: https://lkml.kernel.org/r/c54304d90c777310fb85a3e658d1449173759aab.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla: Change monitored_cpus from char * to cpu_set_t

2023-06-13T20:28:56+00:00

Use a cpumask instead of a char *, reducing memory footprint and code.

No functional change, and in preparation for auto house-keeping.

Link: https://lkml.kernel.org/r/54c46293261d13cb1042d0314486539eeb45fe5d.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla: Add --house-keeping option

2023-06-13T20:26:03+00:00

To avoid having rtla interfering with the measurement threads, add an
option for the user to set the CPUs in which rtla should run. For
instance:

  # rtla timerlat top -H 0 -c 1-7

Will place rtla in the CPU 0, while running the measurement threads in
the CPU 1-7.

Link: https://lkml.kernel.org/r/6a6c78a579a96ba8b02ae67ee1e0ba2cb5e03c4a.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Suggested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)

rtla: Add -C cgroup support

2023-06-13T20:25:13+00:00

The -C option sets a cgroup to the tracer's threads. If the -C option is
passed without arguments, the tracer's thread will inherit rtla's
cgroup. Otherwise, the threads will be placed on the cgroup passed
to the option.

Link: https://lkml.kernel.org/r/cb051477331d292f17c08bf1d66f0e0384bbe5a5.1686066600.git.bristot@kernel.org

Cc: William White 
Cc: Jonathan Corbet 
Tested-by: Juri Lelli 
Signed-off-by: Daniel Bristot de Oliveira 
Signed-off-by: Steven Rostedt (Google)