A bug was reported when rtla-timerlat-top tool performs on-threshold
actions, even though no threshold was hit. This is reproduced even if no
threshold is set at all:

$ rtla timerlat top -q -c 0 --on-threshold shell,command='echo BAD'
BAD
                                     Timer Latency
...

The bug is due to incorrect logic in timerlat_top_bpf_main_loop().
The loop uses timerlat_bpf_wait(), the return values of which are:

- > 0 (number of ringbuffer entries): at least 1 CPU hit threshold
- = 0: time out
- < 0: wait was interrupted by a signal

Commit 3138df6f0cd0 ("rtla/timerlat: Exit top main loop on any non-zero
wait_retval") changed the condition for "threshold hit" from
"wait_reval == 1" (exactly 1 CPU hit threshold) to "wait_retval != 0",
to fix a race where multiple CPUs hit the threshold at the same time.

That also made it incorrectly include a signal (< 0), coming from either
duration expired (SIGALRM) or user interrupt (SIGINT).

Check for wait_retval greater than zero in the if condition to cover all
return values correctly.

Fixes: 3138df6f0cd0 ("rtla/timerlat: Exit top main loop on any non-zero wait_retval")
Reported-by: Attila Fazekas <afazekas@redhat.com>
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260713141047.687877-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

Instead of using getopt_long() directly to parse the command line
arguments given to an RTLA tool, use libsubcmd's parse_options().

Utilizing libsubcmd for parsing command line arguments has several
benefits:

- A help message is automatically generated by libsubcmd from the
  specification, removing the need of writing it by hand.
- Options are sorted into groups based on which part of tracing (CPU,
  thread, auto-analysis, tuning, histogram) they relate to.
- Common parsing patterns for numerical and boolean values now share
  code, with the target variable being stored in the option array.

To avoid duplication of the option parsing logic, RTLA-specific
macros defining struct option values are created:

- RTLA_OPT_* for options common to all tools
- OSNOISE_OPT_* and TIMERLAT_OPT_* for options specific to
  osnoise/timerlat tools
- HIST_OPT_* macros for options specific to histogram-based tools.

Individual *_parse_args() functions then construct an array out of
these macros that is then passed to libsubcmd's parse_options().

All code specific to command line options parsing is moved out of the
individual tool files into a new file, cli.c, which also contains the
contents of the rtla.c file. A private header, cli_p.h, is added
alongside the public header cli.h, so that unit tests are able to test
statically declared option callbacks.

Minor changes:

- The return value of tool-level help option changes to 129, as this is
  the value set by libsubcmd; this is reflected in affected test cases.
  The implementation of help for command-level and tracer-level help
  is set to 129 as well for consistency, and the change is reflected in
  exit value documentation.
- Related to the above, {rtla,osnoise,timerlat}_usage() are marked
  __noreturn and exit() is removed from after they are called for
  cleaner code.
- The error messages for invalid argument for options --dma-latency and
  -E/--entries were corrected, fixing off-by-one in the limits.

Note that unsetting options (using --no-<opt> syntax) is currently not
implemented for options that use custom callbacks. For --irq and
--thread, it will never be implemented, as they conflict with already
existing --no-irq and --no-thread with a different meaning.

Assisted-by: Composer:composer-1.5
Link: https://lore.kernel.org/r/20260528103254.2990068-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

Fix --dump-task to --dump-tasks in timerlat_hist usage string
and getopt_long table for consistency with timerlat_top.

Add missing --dump-tasks to timerlat_top usage synopsis.

Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Fixes: 2091336b9a8b ("rtla/timerlat_hist: Add auto-analysis support")
Link: https://lore.kernel.org/r/20260414185223.65353-1-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

Several functions duplicate the logic for handling threshold actions.
When a threshold is reached, these functions stop the trace, perform
configured actions, and restart the trace if --on-threshold continue
is set.

Create common_threshold_handler() to centralize this shared logic and
avoid code duplication. The function executes the configured threshold
actions and restarts the necessary trace instances when appropriate.

Also add should_continue_tracing() helper to encapsulate the check
for whether tracing should continue after a threshold event, improving
code readability at call sites.

In timerlat_top_bpf_main_loop(), use common_params directly instead
of casting through timerlat_params when only common fields are needed.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-5-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

Most memory allocations in rtla happen during early initialization
before any resources are acquired that would require cleanup. In these
cases, propagating allocation errors just adds complexity without any
benefit. There's nothing to clean up, and the program must exit anyway.

This patch introduces fatal allocation wrappers (calloc_fatal,
reallocarray_fatal, strdup_fatal) that call fatal() on allocation
failure. These wrappers simplify the code by eliminating unnecessary
error propagation paths.

The patch converts early allocations to use these wrappers in
actions_init() and related action functions, osnoise_context_alloc()
and osnoise_init_tool(), trace_instance_init() and trace event
functions, and parameter structure allocations in main functions.

This simplifies the code while maintaining the same behavior: immediate
exit on allocation failure during initialization. Allocations that
require cleanup, such as those in histogram allocation functions with
goto cleanup paths, are left unchanged and continue to return errors.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-2-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

nr_cpus does not change at runtime, so passing it through the macro
argument is unnecessary.

Remove the argument and use the global nr_cpus instead.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20260306194953.2511960-5-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

nr_cpus does not change at runtime, so keeping it in struct members is
unnecessary.

Use the global nr_cpus instead of struct members.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20260306194953.2511960-4-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

nr_cpus does not change at runtime, so passing it through function
arguments is unnecessary.

Use the global nr_cpus instead of propagating it via parameters.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20260306194953.2511960-3-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

sysconf(_SC_NPROCESSORS_CONF) (via get_nprocs_conf) reflects
cpu_possible_mask, which is fixed at boot time, so querying it
repeatedly is unnecessary.

Replace multiple calls to sysconf(_SC_NPROCESSORS_CONF) with a single
global nr_cpus variable initialized once at startup.

`#pragma once` in timerlat_u.h is needed for pre-C23 compilers to avoid
redefinition errors.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20260306194953.2511960-2-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>

In the current implementation, the auto-analysis code for printing the
stack captured in the tracefs buffer of the aa instance stops at the
first encountered address that cannot be resolved into a function
symbol.

This is not always the desired behavior on all platforms; sometimes,
there might be resolvable entries after unresolvable ones, and
sometimes, the user might want to inspect the raw pointers for the
unresolvable entries.

Add a new option, --stack-format, with three values:

- truncate: stop at first unresolvable entry. This is the current
  behavior, and is kept as the default.
- skip: skip unresolvable entries, but do not stop on them.
- full: print all entries, including unresolvable ones.

To make this work, the "size" field of the stack entry is now also read
and used as the maximum number of entries to print, capped at 64, since
that is the fixed length of the "caller" field.

Link: https://lore.kernel.org/r/20260119115222.744150-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>