linux.git/tools/perf/util/scripting-engines, branch v4.19

tools lib traceevent, perf tools: Rename pevent_read_number_* APIs

2018-08-13T18:22:05+00:00

In order to make libtraceevent into a proper library, variables, data
structures and functions require a unique prefix to prevent name space
conflicts. That prefix will be "tep_" and not "pevent_". This changes
APIs: pevent_read_number, pevent_read_number_field

Signed-off-by: Tzvetomir Stoyanov (VMware) 
Cc: Andrew Morton 
Cc: Peter Zijlstra 
Cc: Yordan Karadzhov (VMware) 
Cc: linux-trace-devel@vger.kernel.org
Link: http://lkml.kernel.org/r/20180808180700.804271434@goodmis.org
Signed-off-by: Steven Rostedt 
Signed-off-by: Arnaldo Carvalho de Melo

tools lib traceevent, perf tools: Rename struct pevent to struct tep_handle

2018-08-10T18:29:35+00:00

In order to make libtraceevent into a proper library, variables, data
structures and functions require a unique prefix to prevent name space
conflicts. That prefix will be "tep_" and not "pevent_". This changes
the struct pevent to struct tep_handle.

Signed-off-by: Tzvetomir Stoyanov (VMware) 
Cc: Andrew Morton 
Cc: Peter Zijlstra 
Cc: Yordan Karadzhov (VMware) 
Cc: linux-trace-devel@vger.kernel.org
Link: http://lkml.kernel.org/r/20180808180659.706175783@goodmis.org
Signed-off-by: Steven Rostedt 
Signed-off-by: Arnaldo Carvalho de Melo

perf script python: Fix dict reference counting

2018-07-11T13:45:24+00:00

The dictionaries are attached to the parameter tuple that steals the
references and takes care of releasing them when appropriate.  The code
should not decrement the reference counts explicitly.  E.g. if libpython
has been built with reference debugging enabled, the superfluous DECREFs
will trigger this error when running perf script:

  Fatal Python error: Objects/tupleobject.c:238 object at
  0x7f10f2041b40 has negative ref count -1
  Aborted (core dumped)

If the reference debugging is not enabled, the superfluous DECREFs might
cause the dict objects to be silently released while they are still in
use. This may trigger various other assertions or just cause perf
crashes and/or weird and unexpected data changes in the stored Python
objects.

Signed-off-by: Janne Huttunen 
Acked-by: Jiri Olsa 
Acked-by: Namhyung Kim 
Cc: Alexander Shishkin 
Cc: Andi Kleen 
Cc: Jaroslav Skarvada 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Link: http://lkml.kernel.org/r/1531133990-17485-1-git-send-email-janne.huttunen@nokia.com
Signed-off-by: Arnaldo Carvalho de Melo

perf tools: Generate a Python script compatible with Python 2 and 3

2018-07-11T13:01:50+00:00

When generating a Python script with "perf script -g python", produce
one that is compatible with Python 2 and 3. The difference between the
two generated scripts is:

  --- python2-perf-script.py	2018-05-08 15:35:00.865889705 -0400
  +++ python3-perf-script.py	2018-05-08 15:34:49.019789564 -0400
  @@ -7,6 +7,8 @@
   # be retrieved using Python functions of the form common_*(context).
   # See the perf-script-python Documentation for the list of available functions.

  +from __future__ import print_function
  +
   import os
   import sys

  @@ -18,10 +20,10 @@

   def trace_begin():
  -	print "in trace_begin"
  +	print("in trace_begin")

   def trace_end():
  -	print "in trace_end"
  +	print("in trace_end")

   def raw_syscalls__sys_enter(event_name, context, common_cpu,
   	common_secs, common_nsecs, common_pid, common_comm,
  @@ -29,26 +31,26 @@
   		print_header(event_name, common_cpu, common_secs, common_nsecs,
   			common_pid, common_comm)

  -		print "id=%d, args=%s" % \
  -		(id, args)
  +		print("id=%d, args=%s" % \
  +		(id, args))

  -		print 'Sample: {'+get_dict_as_string(perf_sample_dict['sample'], ', ')+'}'
  +		print('Sample: {'+get_dict_as_string(perf_sample_dict['sample'], ', ')+'}')

   		for node in common_callchain:
   			if 'sym' in node:
  -				print "\t[%x] %s" % (node['ip'], node['sym']['name'])
  +				print("\t[%x] %s" % (node['ip'], node['sym']['name']))
   			else:
  -				print "	[%x]" % (node['ip'])
  +				print("	[%x]" % (node['ip']))

  -		print "\n"
  +		print()

   def trace_unhandled(event_name, context, event_fields_dict, perf_sample_dict):
  -		print get_dict_as_string(event_fields_dict)
  -		print 'Sample: {'+get_dict_as_string(perf_sample_dict['sample'], ', ')+'}'
  +		print(get_dict_as_string(event_fields_dict))
  +		print('Sample: {'+get_dict_as_string(perf_sample_dict['sample'], ', ')+'}')

   def print_header(event_name, cpu, secs, nsecs, pid, comm):
  -	print "%-20s %5u %05u.%09u %8u %-20s " % \
  -	(event_name, cpu, secs, nsecs, pid, comm),
  +	print("%-20s %5u %05u.%09u %8u %-20s " % \
  +	(event_name, cpu, secs, nsecs, pid, comm), end="")

   def get_dict_as_string(a_dict, delimiter=' '):
   	return delimiter.join(['%s=%s'%(k,str(v))for k,v in sorted(a_dict.items())])

Signed-off-by: Jeremy Cline 
Cc: Alexander Shishkin 
Cc: Herton Krzesinski 
Cc: Jiri Olsa 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Link: http://lkml.kernel.org/r/0100016341a7278a-d178c724-2b0f-49ca-be93-80a7d51aaa0d-000000@email.amazonses.com
Signed-off-by: Arnaldo Carvalho de Melo

perf script python: Add more PMU fields to event handler dict

2018-06-06T18:38:26+00:00

When doing pmu sampling and then running a script with perf script -s
script.py, the process_event function gets dictionary with some fields
from the perf ring buffer (like ip, sym, callchain etc).

But we miss quite a few fields we report now, for example, LBRs, data
source, weight, transaction, iregs, uregs, etc.

This patch reports these fields for perf script python processing.

  New keys/items:
  ---------------
  key  : brstack
  items: from, to, from_dsoname, to_dsoname, mispred,
         predicted, in_tx, abort, cycles.

  key  : brstacksym
  items: from, to, pred, in_tx, abort (converted string)

  key  : datasrc
  key  : datasrc_decode (decoded string)
  key  : iregs
  key  : uregs
  key  : weight
  key  : transaction

  v2:
  ---
  Add new fields for dso.
  Use PyBool_FromLong() for mispred/predicted/in_tx/abort

Committer notes:

!sym->name isn't valid, as its not a pointer, its a [0] array, use
!sym->name[0] instead, guaranteed to be the case by symbol__new.

This was caught by just one of the containers:

  52    54.22 ubuntu:17.04                  : FAIL gcc (Ubuntu 6.3.0-12ubuntu2) 6.3.0 20170406

    CC       /tmp/build/perf/util/scripting-engines/trace-event-python.o
  util/scripting-engines/trace-event-python.c:534:20: error: address of array 'sym->name' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
          if (!sym || !sym->name)
                    ~~~~~~^~~~
  1 error generated.
  mv: cannot stat '/tmp/build/perf/util/scripting-engines/.trace-event-python.o.tmp': No such file or directory
  /git/linux/tools/build/Makefile.build:96: recipe for target '/tmp/build/perf/util/scripting-engines/trace-event-python.o' failed
  make[5]: *** [/tmp/build/perf/util/scripting-engines/trace-event-python.o] Error 1

Signed-off-by: Jin Yao 
Reviewed-by: Andi Kleen 
Cc: Alexander Shishkin 
Cc: Jin Yao 
Cc: Jiri Olsa 
Cc: Kan Liang 
Cc: Peter Zijlstra 
Link: http://lkml.kernel.org/r/1527843663-32288-3-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo

perf script python: Move dsoname code to a new function

2018-06-06T15:52:09+00:00

This patch creates a new function get_dsoname() and move the code which
gets the dsoname string to this function.

That's because in next patch, when we process LBR data, we will also
need get_dsoname() to return dsoname for branch from/to.

Signed-off-by: Jin Yao 
Reviewed-by: Andi Kleen 
Cc: Alexander Shishkin 
Cc: Jiri Olsa 
Cc: Kan Liang 
Cc: Peter Zijlstra 
Link: http://lkml.kernel.org/r/1527843663-32288-2-git-send-email-yao.jin@linux.intel.com
Signed-off-by: Arnaldo Carvalho de Melo

perf script python: Add addr into perf sample dict

2018-05-30T18:39:31+00:00

ARM CoreSight auxtrace uses 'sample->addr' to record the target address
for branch instructions, so the data of 'sample->addr' is required for
tracing data analysis.

This commit collects data of 'sample->addr' into perf sample dict,
finally can be used for python script for parsing event.

Signed-off-by: Leo Yan 
Cc: Alexander Shishkin 
Cc: Jiri Olsa 
Cc: Jonathan Corbet 
Cc: Mathieu Poirier 
Cc: Mike Leach 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Cc: Robert Walker 
Cc: Tor Jeremiassen 
Cc: coresight@lists.linaro.org
Cc: kim.phillips@arm.co
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-doc@vger.kernel.org
Link: http://lkml.kernel.org/r/1527497103-3593-3-git-send-email-leo.yan@linaro.org
Signed-off-by: Arnaldo Carvalho de Melo

perf tools: Add Python 3 support

2018-02-19T15:28:23+00:00

Added Python 3 support while keeping Python 2.7 compatibility.

Committer notes:

This doesn't make it to auto detect python 3, one has to explicitely ask
it to build with python 3 devel files, here are the instructions
provided by Jaroslav:

 ---
  $ cp -a tools/perf tools/python3-perf
  $ make V=1 prefix=/usr -C tools/perf PYTHON=/usr/bin/python2 all
  $ make V=1 prefix=/usr -C tools/python3-perf PYTHON=/usr/bin/python3 all
  $ make V=1 prefix=/usr -C tools/python3-perf PYTHON=/usr/bin/python3 DESTDIR=%{buildroot} install-python_ext
  $ make V=1 prefix=/usr -C tools/perf PYTHON=/usr/bin/python2 DESTDIR=%{buildroot} install-python_ext
 ---

We need to make this automatic, just like the existing tests for checking if
the python2 devel files are in place, allowing the build with python3 if
available, fallbacking to python2 and then just disabling it if none are
available.

So, using the PYTHON variable to build it using O= we get:

Before this patch:

  $ rpm -q python3 python3-devel
  python3-3.6.4-7.fc27.x86_64
  python3-devel-3.6.4-7.fc27.x86_64
  $ rm -rf /tmp/build/perf/ ; mkdir -p /tmp/build/perf ; make O=/tmp/build/perf PYTHON=/usr/bin/python3 -C tools/perf install-bin
  make: Entering directory '/home/acme/git/linux/tools/perf'
  
  Makefile.config:670: Python 3 is not yet supported; please set
  Makefile.config:671: PYTHON and/or PYTHON_CONFIG appropriately.
  Makefile.config:672: If you also have Python 2 installed, then
  Makefile.config:673: try something like:
  Makefile.config:674:
  Makefile.config:675:   make PYTHON=python2
  Makefile.config:676:
  Makefile.config:677: Otherwise, disable Python support entirely:
  Makefile.config:678:
  Makefile.config:679:   make NO_LIBPYTHON=1
  Makefile.config:680:
  Makefile.config:681: *** .  Stop.
  make[1]: *** [Makefile.perf:212: sub-make] Error 2
  make: *** [Makefile:110: install-bin] Error 2
  make: Leaving directory '/home/acme/git/linux/tools/perf'
  $

After:

  $ make O=/tmp/build/perf PYTHON=python3 -C tools/perf install-bin
  $ ldd ~/bin/perf | grep python
	libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007f58a31e8000)
  $ rpm -qf /lib64/libpython3.6m.so.1.0
  python3-libs-3.6.4-7.fc27.x86_64
  $

Now verify that when using the binding the right ELF file is loaded,
using perf trace:

  $ perf trace -e open* perf test python
     0.051 ( 0.016 ms): perf/3927 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC           ) = 3

  18: 'import perf' in python                               :
     8.849 ( 0.013 ms): sh/3929 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC           ) = 3

    25.572 ( 0.008 ms): python3/3931 openat(dfd: CWD, filename: /tmp/build/perf/python/perf.cpython-36m-x86_64-linux-gnu.so, flags: CLOEXEC) = 3

 Ok

  $

And using tools/perf/python/twatch.py, to show PERF_RECORD_ metaevents:

  $ python3 tools/perf/python/twatch.py
  cpu: 3, pid: 16060, tid: 16060 { type: fork, pid: 5207, ppid: 16060, tid: 5207, ptid: 16060, time: 10798513015459}
  cpu: 3, pid: 16060, tid: 16060 { type: fork, pid: 5208, ppid: 16060, tid: 5208, ptid: 16060, time: 10798513562503}
  cpu: 0, pid: 5208, tid: 5208 { type: comm, pid: 5208, tid: 5208, comm: grep }
  cpu: 2, pid: 5207, tid: 5207 { type: comm, pid: 5207, tid: 5207, comm: ps }
  cpu: 2, pid: 5207, tid: 5207 { type: exit, pid: 5207, ppid: 5207, tid: 5207, ptid: 5207, time: 10798551337484}
  cpu: 3, pid: 5208, tid: 5208 { type: exit, pid: 5208, ppid: 5208, tid: 5208, ptid: 5208, time: 10798551292153}
  cpu: 3, pid: 601, tid: 601 { type: fork, pid: 5209, ppid: 601, tid: 5209, ptid: 601, time: 10801779977324}
  ^CTraceback (most recent call last):
    File "tools/perf/python/twatch.py", line 68, in 
      main()
    File "tools/perf/python/twatch.py", line 40, in main
      evlist.poll(timeout = -1)
  KeyboardInterrupt
  $

  # ps ax|grep twatch
 5197 pts/8    S+     0:00 python3 tools/perf/python/twatch.py
  # ls -la /proc/5197/smaps
  -r--r--r--. 1 acme acme 0 Feb 19 13:14 /proc/5197/smaps
  # grep python /proc/5197/smaps
  558111307000-558111309000 r-xp 00000000 fd:00 3151710  /usr/bin/python3.6
  558111508000-558111509000 r--p 00001000 fd:00 3151710  /usr/bin/python3.6
  558111509000-55811150a000 rw-p 00002000 fd:00 3151710  /usr/bin/python3.6
  7ffad6fc1000-7ffad7008000 r-xp 00000000 00:2d 220196   /tmp/build/perf/python/perf.cpython-36m-x86_64-linux-gnu.so
  7ffad7008000-7ffad7207000 ---p 00047000 00:2d 220196   /tmp/build/perf/python/perf.cpython-36m-x86_64-linux-gnu.so
  7ffad7207000-7ffad7208000 r--p 00046000 00:2d 220196   /tmp/build/perf/python/perf.cpython-36m-x86_64-linux-gnu.so
  7ffad7208000-7ffad7215000 rw-p 00047000 00:2d 220196   /tmp/build/perf/python/perf.cpython-36m-x86_64-linux-gnu.so
  7ffadea77000-7ffaded3d000 r-xp 00000000 fd:00 3151795  /usr/lib64/libpython3.6m.so.1.0
  7ffaded3d000-7ffadef3c000 ---p 002c6000 fd:00 3151795  /usr/lib64/libpython3.6m.so.1.0
  7ffadef3c000-7ffadef42000 r--p 002c5000 fd:00 3151795  /usr/lib64/libpython3.6m.so.1.0
  7ffadef42000-7ffadefa5000 rw-p 002cb000 fd:00 3151795  /usr/lib64/libpython3.6m.so.1.0
  #

And with this patch, but building normally, without specifying the
PYTHON=python3 part, which will make it use python2 if its devel files are
available, like in this test:

  $ make O=/tmp/build/perf -C tools/perf install-bin
  $ ldd ~/bin/perf | grep python
	libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f6a44410000)
  $ ldd /tmp/build/perf/python_ext_build/lib/perf.so  | grep python
	libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fed28a2c000)
  $

  [acme@jouet perf]$ tools/perf/python/twatch.py
  cpu: 0, pid: 2817, tid: 2817 { type: fork, pid: 2817, ppid: 2817, tid: 8910, ptid: 2817, time: 11126454335306}
  cpu: 0, pid: 2817, tid: 2817 { type: comm, pid: 2817, tid: 8910, comm: worker }
  $ ps ax | grep twatch.py
   8909 pts/8    S+     0:00 /usr/bin/python tools/perf/python/twatch.py
  $ grep python /proc/8909/smaps
  5579de658000-5579de659000 r-xp 00000000 fd:00 3156044  /usr/bin/python2.7
  5579de858000-5579de859000 r--p 00000000 fd:00 3156044  /usr/bin/python2.7
  5579de859000-5579de85a000 rw-p 00001000 fd:00 3156044  /usr/bin/python2.7
  7f0de01f7000-7f0de023e000 r-xp 00000000 00:2d 230695   /tmp/build/perf/python/perf.so
  7f0de023e000-7f0de043d000 ---p 00047000 00:2d 230695   /tmp/build/perf/python/perf.so
  7f0de043d000-7f0de043e000 r--p 00046000 00:2d 230695   /tmp/build/perf/python/perf.so
  7f0de043e000-7f0de044b000 rw-p 00047000 00:2d 230695   /tmp/build/perf/python/perf.so
  7f0de6f0f000-7f0de6f13000 r-xp 00000000 fd:00 134975   /usr/lib64/python2.7/lib-dynload/_localemodule.so
  7f0de6f13000-7f0de7113000 ---p 00004000 fd:00 134975   /usr/lib64/python2.7/lib-dynload/_localemodule.so
  7f0de7113000-7f0de7114000 r--p 00004000 fd:00 134975   /usr/lib64/python2.7/lib-dynload/_localemodule.so
  7f0de7114000-7f0de7115000 rw-p 00005000 fd:00 134975   /usr/lib64/python2.7/lib-dynload/_localemodule.so
  7f0de7e73000-7f0de8052000 r-xp 00000000 fd:00 3173292  /usr/lib64/libpython2.7.so.1.0
  7f0de8052000-7f0de8251000 ---p 001df000 fd:00 3173292  /usr/lib64/libpython2.7.so.1.0
  7f0de8251000-7f0de8255000 r--p 001de000 fd:00 3173292  /usr/lib64/libpython2.7.so.1.0
  7f0de8255000-7f0de8291000 rw-p 001e2000 fd:00 3173292  /usr/lib64/libpython2.7.so.1.0
  $

Signed-off-by: Jaroslav Škarvada 
Tested-by: Arnaldo Carvalho de Melo 
Cc: Adrian Hunter 
Cc: David Ahern 
Cc: Jiri Olsa 
Cc: Namhyung Kim 
Cc: Wang Nan 
LPU-Reference: 20180119205641.24242-1-jskarvad@redhat.com
Link: https://lkml.kernel.org/n/tip-8d7dt9kqp83vsz25hagug8fu@git.kernel.org
[ Removed explicit check for python version, allowing it to really build with python3 ]
Signed-off-by: Arnaldo Carvalho de Melo

perf script python: Add script to profile and resolve physical mem type

2018-01-12T14:06:57+00:00

There could be different types of memory in the system. E.g normal
System Memory, Persistent Memory. To understand how the workload maps to
those memories, it's important to know the I/O statistics of them.  Perf
can collect physical addresses, but those are raw data.  It still needs
extra work to resolve the physical addresses.  Provide a script to
facilitate the physical addresses resolving and I/O statistics.

Profile with MEM_INST_RETIRED.ALL_LOADS or MEM_UOPS_RETIRED.ALL_LOADS
event if any of them is available.

Look up the /proc/iomem and resolve the physical address.  Provide
memory type summary.

Here is an example output:

  # perf script report mem-phys-addr
  Event: mem_inst_retired.all_loads:P
  Memory type                                    count   percentage
  ----------------------------------------  -----------  -----------
  System RAM                                        74        53.2%
  Persistent Memory                                 55        39.6%
  N/A

  ---

Changes since V2:
 - Apply the new license rules.
 - Add comments for globals

Changes since V1:
 - Do not mix DLA and Load Latency. Do not compare the loads and stores.
   Only profile the loads.
 - Use event name to replace the RAW event

Signed-off-by: Kan Liang 
Reviewed-by: Andi Kleen 
Cc: Dan Williams 
Cc: Jiri Olsa 
Cc: Peter Zijlstra 
Cc: Philippe Ombredanne 
Cc: Stephane Eranian 
Link: https://lkml.kernel.org/r/1515099595-34770-1-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo

perf perf: Remove duplicate includes

2017-12-27T15:15:49+00:00

These duplicate includes have been found with scripts/checkincludes.pl
but they have been removed manually to avoid removing false positives.

Signed-off-by: Pravin Shedge 
Cc: David S. Miller 
Cc: Greg Kroah-Hartman 
Cc: Jiri Olsa 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Link: http://lkml.kernel.org/r/1512582204-6493-1-git-send-email-pravin.shedge4linux@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo