linux-stable.git/tools/build, branch v5.12.2

perf build: Move feature cleanup under tools/build

2021-03-06T19:54:24+00:00

Arnaldo reported issue for following build command:

  $ rm -rf /tmp/krava; mkdir /tmp/krava; make O=/tmp/krava clean
    CLEAN    config
  /bin/sh: line 0: cd: /tmp/krava/feature/: No such file or directory
  ../../scripts/Makefile.include:17: *** output directory "/tmp/krava/feature/" does not exist.  Stop.
  make[1]: *** [Makefile.perf:1010: config-clean] Error 2
  make: *** [Makefile:90: clean] Error 2

The problem is that now that we include scripts/Makefile.include
in feature's Makefile (which is fine and needed), we need to ensure
the OUTPUT directory exists, before executing (out of tree) clean
command.

Removing the feature's cleanup from perf Makefile and fixing
feature's cleanup under build Makefile, so it now checks that
there's existing OUTPUT directory before calling the clean.

Fixes: 211a741cd3e1 ("tools: Factor Clang, LLC and LLVM utils definitions")
Reported-by: Arnaldo Carvalho de Melo 
Signed-off-by: Jiri Olsa 
Tested-by: Arnaldo Carvalho de Melo 
Tested-by: Sedat Dilek  # LLVM/Clang v13-git
Cc: Alexander Shishkin 
Cc: Ian Rogers 
Cc: Mark Rutland 
Cc: Michael Petlan 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Link: http://lore.kernel.org/lkml/20210224150831.409639-1-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo

Merge tag 'perf-tools-for-v5.12-2020-02-19' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux

2021-02-22T21:59:43+00:00

Pull perf tool updates from Arnaldo Carvalho de Melo:
"New features:

- Support instruction latency in 'perf report', with both memory
latency (weight) and instruction latency information, users can
locate expensive load instructions and understand time spent in
different stages.

- Extend 'perf c2c' to display the number of loads which were blocked
by data or address conflict.

- Add 'perf stat' support for L2 topdown events in systems such as
Intel's Sapphire rapids server.

- Add support for PERF_SAMPLE_CODE_PAGE_SIZE in various tools, as a
sort key, for instance:

perf report --stdio --sort=comm,symbol,code_page_size

- New 'perf daemon' command to run long running sessions while
providing a way to control the enablement of events without
restarting a traditional 'perf record' session.

- Enable counting events for BPF programs in 'perf stat' just like
for other targets (tid, cgroup, cpu, etc), e.g.:

# perf stat -e ref-cycles,cycles -b 254 -I 1000
1.487903822 115,200 ref-cycles
1.487903822 86,012 cycles
2.489147029 80,560 ref-cycles
2.489147029 73,784 cycles
^C

The example above counts 'cycles' and 'ref-cycles' of BPF program
of id 254. It is similar to bpftool-prog-profile command, but more
flexible.

- Support the new layout for PERF_RECORD_MMAP2 to carry the DSO
build-id using infrastructure generalised from the eBPF subsystem,
removing the need for traversing the perf.data file to collect
build-ids at the end of 'perf record' sessions and helping with
long running sessions where binaries can get replaced in updates,
leading to possible mis-resolution of symbols.

- Support filtering by hex address in 'perf script'.

- Support DSO filter in 'perf script', like in other perf tools.

- Add namespaces support to 'perf inject'

- Add support for SDT (Dtrace Style Markers) events on ARM64.

perf record:

- Fix handling of eventfd() when draining a buffer in 'perf record'.

- Improvements to the generation of metadata events for pre-existing
threads (mmaps, comm, etc), speeding up the work done at the start
of system wide or per CPU 'perf record' sessions.

Hardware tracing:

- Initial support for tracing KVM with Intel PT.

- Intel PT fixes for IPC

- Support Intel PT PSB (synchronization packets) events.

- Automatically group aux-output events to overcome --filter syntax.

- Enable PERF_SAMPLE_DATA_SRC on ARMs SPE.

- Update ARM's CoreSight hardware tracing OpenCSD library to v1.0.0.

perf annotate TUI:

- Fix handling of 'k' ("show line number") hotkey

- Fix jump parsing for C++ code.

perf probe:

- Add protection to avoid endless loop.

cgroups:

- Avoid reading cgroup mountpoint multiple times, caching it.

- Fix handling of cgroup v1/v2 in mixed hierarchy.

Symbol resolving:

- Add OCaml symbol demangling.

- Further fixes for handling PE executables when using perf with Wine
and .exe/.dll files.

- Fix 'perf unwind' DSO handling.

- Resolve symbols against debug file first, to deal with artifacts
related to LTO.

- Fix gap between kernel end and module start on powerpc.

Reporting tools:

- The DSO filter shouldn't show samples in unresolved maps.

- Improve debuginfod support in various tools.

build ids:

- Fix 16-byte build ids in 'perf buildid-cache', add a 'perf test'
entry for that case.

perf test:

- Support for PERF_SAMPLE_WEIGHT_STRUCT.

- Add test case for PERF_SAMPLE_CODE_PAGE_SIZE.

- Shell based tests for 'perf daemon's commands ('start', 'stop,
'reconfig', 'list', etc).

- ARM cs-etm 'perf test' fixes.

- Add parse-metric memory bandwidth testcase.

Compiler related:

- Fix 'perf probe' kretprobe issue caused by gcc 11 bug when used
with -fpatchable-function-entry.

- Fix ARM64 build with gcc 11's -Wformat-overflow.

- Fix unaligned access in sample parsing test.

- Fix printf conversion specifier for IP addresses on arm64, s390 and
powerpc.

Arch specific:

- Support exposing Performance Monitor Counter SPRs as part of
extended regs on powerpc.

- Add JSON 'perf stat' metrics for ARM64's imx8mp, imx8mq and imx8mn
DDR, fix imx8mm ones.

- Fix common and uarch events for ARM64's A76 and Ampere eMag"

* tag 'perf-tools-for-v5.12-2020-02-19' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux: (148 commits)
perf buildid-cache: Don't skip 16-byte build-ids
perf buildid-cache: Add test for 16-byte build-id
perf symbol: Remove redundant libbfd checks
perf test: Output the sub testing result in cs-etm
perf test: Suppress logs in cs-etm testing
perf tools: Fix arm64 build error with gcc-11
perf intel-pt: Add documentation for tracing virtual machines
perf intel-pt: Split VM-Entry and VM-Exit branches
perf intel-pt: Adjust sample flags for VM-Exit
perf intel-pt: Allow for a guest kernel address filter
perf intel-pt: Support decoding of guest kernel
perf machine: Factor out machine__idle_thread()
perf machine: Factor out machines__find_guest()
perf intel-pt: Amend decoder to track the NR flag
perf intel-pt: Retain the last PIP packet payload as is
perf intel_pt: Add vmlaunch and vmresume as branches
perf script: Add branch types for VM-Entry and VM-Exit
perf auxtrace: Automatically group aux-output events
perf test: Fix unaligned access in sample parsing test
perf tools: Support arch specific PERF_SAMPLE_WEIGHT_STRUCT processing
...

tools: Factor Clang, LLC and LLVM utils definitions

2021-01-29T00:25:34+00:00

When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.

While looking into the source code I found duplicate assignments in misc tools
for the LLVM eco system, e.g. clang and llvm-objcopy.

Move the Clang, LLC and/or LLVM utils definitions to tools/scripts/Makefile.include
file and add missing includes where needed. Honestly, I was inspired by the commit
c8a950d0d3b9 ("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").

I tested with bpftool and perf on Debian/testing AMD64 and LLVM/Clang v11.1.0-rc1.

Build instructions:

[ make and make-options ]
MAKE="make V=1"
MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"

[ clean-up ]
$MAKE $MAKE_OPTS -C tools/ clean

[ bpftool ]
$MAKE $MAKE_OPTS -C tools/bpf/bpftool/

[ perf ]
PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/

I was careful with respecting the user's wish to override custom compiler, linker,
GNU/binutils and/or LLVM utils settings.

Signed-off-by: Sedat Dilek 
Signed-off-by: Daniel Borkmann 
Acked-by: Andrii Nakryiko 
Acked-by: Jiri Olsa  # tools/build and tools/perf
Link: https://lore.kernel.org/bpf/20210128015117.20515-1-sedat.dilek@gmail.com

perf cs-etm: Update ARM's CoreSight hardware tracing OpenCSD library to v1.0.0

2021-01-20T17:34:20+00:00

Replace the OCSD_INSTR switch statement with an if to fix compilation
error about unhandled values and avoid this issue again in the future.

Add new OCSD_GEN_TRC_ELEM_SYNC_MARKER and OCSD_GEN_TRC_ELEM_MEMTRANS
enum values to fix unhandled value compilation error. Currently they are
ignored.

Increase the minimum version number to v1.0.0 now that new enum values
are used that are only present in this version.

Signed-off-by: James Clark 
Reviewed-by: Mathieu Poirier 
Reviewed-by: Mike Leach 
Tested-by: Mike Leach 
Cc: Alexander Shishkin 
Cc: Al Grant 
Cc: Ingo Molnar 
Cc: Jiri Olsa 
Cc: John Garry 
Cc: Leo Yan 
Cc: Mark Rutland 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Cc: Suzuki K Poulose 
Cc: Will Deacon 
Link: https://lore.kernel.org/r/20210108142752.27872-1-james.clark@arm.com
Signed-off-by: Arnaldo Carvalho de Melo

perf build: Support build BPF skeletons with perf

2021-01-15T18:49:07+00:00

BPF programs are useful in perf to profile BPF programs.

BPF skeleton is by far the easiest way to write BPF tools. Enable
building BPF skeletons in util/bpf_skel. A dummy bpf skeleton is added.
More bpf skeletons will be added for different use cases.

Signed-off-by: Song Liu 
Tested-by: Arnaldo Carvalho de Melo 
Acked-by: Jiri Olsa 
Acked-by: Namhyung Kim 
Cc: Alexander Shishkin 
Cc: Mark Rutland 
Cc: Peter Zijlstra 
Cc: kernel-team@fb.com
Link: http://lore.kernel.org/lkml/20201229214214.3413833-3-songliubraving@fb.com
Signed-off-by: Arnaldo Carvalho de Melo

Merge remote-tracking branch 'torvalds/master' into perf/core

2020-12-17T17:37:24+00:00

To pick up fixes and check what UAPI headers need to be synched.

Signed-off-by: Arnaldo Carvalho de Melo

tools build: Add missing libcap to test-all.bin target

2020-12-17T17:36:16+00:00

We're missing -lcap in test-all.bin target, so in case it's the only
library missing (if more are missing test-all.bin fails anyway), we will
falsely claim that we detected it and fail build, like:

  $ make
  ...
  Auto-detecting system features:
  ...                         dwarf: [ on  ]
  ...            dwarf_getlocations: [ on  ]
  ...                         glibc: [ on  ]
  ...                        libbfd: [ on  ]
  ...                libbfd-buildid: [ on  ]
  ...                        libcap: [ on  ]
  ...                        libelf: [ on  ]
  ...                       libnuma: [ on  ]
  ...        numa_num_possible_cpus: [ on  ]
  ...                       libperl: [ on  ]
  ...                     libpython: [ on  ]
  ...                     libcrypto: [ on  ]
  ...                     libunwind: [ on  ]
  ...            libdw-dwarf-unwind: [ on  ]
  ...                          zlib: [ on  ]
  ...                          lzma: [ on  ]
  ...                     get_cpuid: [ on  ]
  ...                           bpf: [ on  ]
  ...                        libaio: [ on  ]
  ...                       libzstd: [ on  ]
  ...        disassembler-four-args: [ on  ]

  ...

    CC       builtin-ftrace.o

  In file included from builtin-ftrace.c:29:
  util/cap.h:11:10: fatal error: sys/capability.h: No such file or directory
     11 | #include 
        |          ^~~~~~~~~~~~~~~~~~
  compilation terminated.

Fixes: 74d5f3d06f707eb5 ("tools build: Add capability-related feature detection")
Signed-off-by: Jiri Olsa 
Cc: Alexander Shishkin 
Cc: Ian Rogers 
Cc: Igor Lubashev 
Cc: Mark Rutland 
Cc: Michael Petlan 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Cc: Stephane Eranian 
Link: http://lore.kernel.org/lkml/20201203230836.3751981-1-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo

Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

2020-11-14T17:13:41+00:00

Daniel Borkmann says:

====================
pull-request: bpf-next 2020-11-14

1) Add BTF generation for kernel modules and extend BTF infra in kernel
   e.g. support for split BTF loading and validation, from Andrii Nakryiko.

2) Support for pointers beyond pkt_end to recognize LLVM generated patterns
   on inlined branch conditions, from Alexei Starovoitov.

3) Implements bpf_local_storage for task_struct for BPF LSM, from KP Singh.

4) Enable FENTRY/FEXIT/RAW_TP tracing program to use the bpf_sk_storage
   infra, from Martin KaFai Lau.

5) Add XDP bulk APIs that introduce a defer/flush mechanism to optimize the
   XDP_REDIRECT path, from Lorenzo Bianconi.

6) Fix a potential (although rather theoretical) deadlock of hashtab in NMI
   context, from Song Liu.

7) Fixes for cross and out-of-tree build of bpftool and runqslower allowing build
   for different target archs on same source tree, from Jean-Philippe Brucker.

8) Fix error path in htab_map_alloc() triggered from syzbot, from Eric Dumazet.

9) Move functionality from test_tcpbpf_user into the test_progs framework so it
   can run in BPF CI, from Alexander Duyck.

10) Lift hashtab key_size limit to be larger than MAX_BPF_STACK, from Florian Lehner.

Note that for the fix from Song we have seen a sparse report on context
imbalance which requires changes in sparse itself for proper annotation
detection where this is currently being discussed on linux-sparse among
developers [0]. Once we have more clarification/guidance after their fix,
Song will follow-up.

  [0] https://lore.kernel.org/linux-sparse/CAHk-=wh4bx8A8dHnX612MsDO13st6uzAz1mJ1PaHHVevJx_ZCw@mail.gmail.com/T/
      https://lore.kernel.org/linux-sparse/20201109221345.uklbp3lzgq6g42zb@ltop.local/T/

* git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (66 commits)
  net: mlx5: Add xdp tx return bulking support
  net: mvpp2: Add xdp tx return bulking support
  net: mvneta: Add xdp tx return bulking support
  net: page_pool: Add bulk support for ptr_ring
  net: xdp: Introduce bulking for xdp tx return path
  bpf: Expose bpf_d_path helper to sleepable LSM hooks
  bpf: Augment the set of sleepable LSM hooks
  bpf: selftest: Use bpf_sk_storage in FENTRY/FEXIT/RAW_TP
  bpf: Allow using bpf_sk_storage in FENTRY/FEXIT/RAW_TP
  bpf: Rename some functions in bpf_sk_storage
  bpf: Folding omem_charge() into sk_storage_charge()
  selftests/bpf: Add asm tests for pkt vs pkt_end comparison.
  selftests/bpf: Add skb_pkt_end test
  bpf: Support for pointers beyond pkt_end.
  tools/bpf: Always run the *-clean recipes
  tools/bpf: Add bootstrap/ to .gitignore
  bpf: Fix NULL dereference in bpf_task_storage
  tools/bpftool: Fix build slowdown
  tools/runqslower: Build bpftool using HOSTCC
  tools/runqslower: Enable out-of-tree build
  ...
====================

Link: https://lore.kernel.org/r/20201114020819.29584-1-daniel@iogearbox.net
Signed-off-by: Jakub Kicinski

tools: Factor HOSTCC, HOSTLD, HOSTAR definitions

2020-11-11T20:18:22+00:00

Several Makefiles in tools/ need to define the host toolchain variables.
Move their definition to tools/scripts/Makefile.include

Signed-off-by: Jean-Philippe Brucker 
Signed-off-by: Andrii Nakryiko 
Acked-by: Jiri Olsa 
Acked-by: Rafael J. Wysocki 
Link: https://lore.kernel.org/bpf/20201110164310.2600671-2-jean-philippe@linaro.org

tools feature: Fixup fast path feature detection

2020-11-03T12:24:20+00:00

22dd1ac91a776752 ("tools: Remove feature-libelf-mmap feature detection")
correctly simplified the this feature detection, but forgot to remove
the call to the removed function in the main() function for the
test-all.c fast path feature detection, making it fail and thus do all
the feature detection individually, fix it.

  $ cat /tmp/build/perf/feature/test-all.make.output
  test-all.c: In function ‘main’:
  test-all.c:188:2: error: implicit declaration of function ‘main_test_libelf_mmap’; did you mean ‘main_test_libelf’? [-Werror=implicit-function-declaration]
    188 |  main_test_libelf_mmap();
        |  ^~~~~~~~~~~~~~~~~~~~~
        |  main_test_libelf
  cc1: all warnings being treated as errors
  $ vim tools/build/feature/test-all.c
  $ rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf ;make V=1 -k O=/tmp/build/perf  -C tools/perf install-bin ; perf test python
  
  $ cat /tmp/build/perf/feature/test-all.make.output
  $

Fixes: 22dd1ac91a776752 ("tools: Remove feature-libelf-mmap feature detection")
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Signed-off-by: Arnaldo Carvalho de Melo