linux.git/tools/perf/Documentation, branch v4.7-rc2 Linux kernel source tree perf report: Add srcline_from/to branch sort keys 2016-05-23T14:25:16+00:00 Andi Kleen ak@linux.intel.com 2016-05-20T20:15:08+00:00 508be0dfe6287d4e6452f5a1dc08856df74cb217 Add "srcline_from" and "srcline_to" branch sort keys that allow to show the source lines of a branch. That makes it much easier to track down where particular branches happen in the program, for example to examine branch mispredictions, or to associate it with cycle counts: % perf record -b -e cycles:p ./tcall % perf report --sort srcline_from,srcline_to,mispredict ... 15.10% tcall.c:18 tcall.c:10 N 14.83% tcall.c:11 tcall.c:5 N 14.12% tcall.c:7 tcall.c:12 N 14.04% tcall.c:12 tcall.c:5 N 12.42% tcall.c:17 tcall.c:18 N 12.39% tcall.c:7 tcall.c:13 N 12.27% tcall.c:13 tcall.c:17 N ... % perf report --sort srcline_from,srcline_to,cycles ... 17.12% tcall.c:18 tcall.c:11 1 17.01% tcall.c:12 tcall.c:6 1 16.98% tcall.c:11 tcall.c:6 1 15.91% tcall.c:17 tcall.c:18 1 6.38% tcall.c:7 tcall.c:17 7 4.80% tcall.c:7 tcall.c:12 8 4.21% tcall.c:7 tcall.c:17 8 2.67% tcall.c:7 tcall.c:12 7 2.62% tcall.c:7 tcall.c:12 10 2.10% tcall.c:7 tcall.c:17 9 1.58% tcall.c:7 tcall.c:12 6 1.44% tcall.c:7 tcall.c:12 5 1.38% tcall.c:7 tcall.c:12 9 1.06% tcall.c:7 tcall.c:17 13 1.05% tcall.c:7 tcall.c:12 4 1.01% tcall.c:7 tcall.c:17 6 Open issues: - Some kernel symbols get misresolved. Signed-off-by: Andi Kleen <ak@linux.intel.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Link: http://lkml.kernel.org/r/1463775308-32748-1-git-send-email-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
Add "srcline_from" and "srcline_to" branch sort keys that allow to show
the source lines of a branch.

That makes it much easier to track down where particular branches happen
in the program, for example to examine branch mispredictions, or to
associate it with cycle counts:

  % perf record -b -e cycles:p ./tcall
  % perf report --sort srcline_from,srcline_to,mispredict
  ...
    15.10%  tcall.c:18       tcall.c:10       N
    14.83%  tcall.c:11       tcall.c:5        N
    14.12%  tcall.c:7        tcall.c:12       N
    14.04%  tcall.c:12       tcall.c:5        N
    12.42%  tcall.c:17       tcall.c:18       N
    12.39%  tcall.c:7        tcall.c:13       N
    12.27%  tcall.c:13       tcall.c:17       N
  ...

  % perf report --sort srcline_from,srcline_to,cycles
  ...
    17.12%  tcall.c:18       tcall.c:11       1
    17.01%  tcall.c:12       tcall.c:6        1
    16.98%  tcall.c:11       tcall.c:6        1
    15.91%  tcall.c:17       tcall.c:18       1
     6.38%  tcall.c:7        tcall.c:17       7
     4.80%  tcall.c:7        tcall.c:12       8
     4.21%  tcall.c:7        tcall.c:17       8
     2.67%  tcall.c:7        tcall.c:12       7
     2.62%  tcall.c:7        tcall.c:12       10
     2.10%  tcall.c:7        tcall.c:17       9
     1.58%  tcall.c:7        tcall.c:12       6
     1.44%  tcall.c:7        tcall.c:12       5
     1.38%  tcall.c:7        tcall.c:12       9
     1.06%  tcall.c:7        tcall.c:17       13
     1.05%  tcall.c:7        tcall.c:12       4
     1.01%  tcall.c:7        tcall.c:17       6

Open issues:

- Some kernel symbols get misresolved.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Link: http://lkml.kernel.org/r/1463775308-32748-1-git-send-email-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf tools: Fix usage of max_stack sysctl 2016-05-20T14:43:56+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-05-19T14:34:06+00:00 fe176085a4d45fb241c04beea872ecbb8bef6a55 We cannot limit processing stacks from the current value of the sysctl, as we may be processing perf.data files, possibly from other machines. Instead use the old PERF_MAX_STACK_DEPTH, the sysctl default, that can be overriden using --max-stack or equivalent. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Brendan Gregg <brendan.d.gregg@gmail.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: He Kuang <hekuang@huawei.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vince Weaver <vincent.weaver@maine.edu> Cc: Wang Nan <wangnan0@huawei.com> Cc: Zefan Li <lizefan@huawei.com> Fixes: 4cb93446c587 ("perf tools: Set the maximum allowed stack from /proc/sys/kernel/perf_event_max_stack") Link: http://lkml.kernel.org/n/tip-eqeutsr7n7wy0c36z24ytvii@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
We cannot limit processing stacks from the current value of the sysctl,
as we may be processing perf.data files, possibly from other machines.

Instead use the old PERF_MAX_STACK_DEPTH, the sysctl default, that can
be overriden using --max-stack or equivalent.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vince Weaver <vincent.weaver@maine.edu>
Cc: Wang Nan <wangnan0@huawei.com>
Cc: Zefan Li <lizefan@huawei.com>
Fixes: 4cb93446c587 ("perf tools: Set the maximum allowed stack from /proc/sys/kernel/perf_event_max_stack")
Link: http://lkml.kernel.org/n/tip-eqeutsr7n7wy0c36z24ytvii@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf record: Disable buildid cache options by default in switch output mode 2016-04-28T12:58:59+00:00 Wang Nan wangnan0@huawei.com 2016-04-20T18:59:52+00:00 0c1d46a8796e8309f1ca693e5cad6f318e4b8159 The cost of buildid cache processing is high: reading all events in output perf.data, opening each elf file to read buildids then copying them into ~/.debug directory. In switch output mode, these heavy works block perf from receiving perf events for too long. Enable no-buildid and no-buildid-cache by default if --switch-output is provided. Still allow user use --no-no-buildid to explicitly enable buildid in this case. Signed-off-by: Wang Nan <wangnan0@huawei.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Zefan Li <lizefan@huawei.com> Cc: pi3orama@163.com Link: http://lkml.kernel.org/r/1461178794-40467-6-git-send-email-wangnan0@huawei.com Signed-off-by: He Kuang <hekuang@huawei.com> [ Updated man page ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
The cost of buildid cache processing is high: reading all events in
output perf.data, opening each elf file to read buildids then copying
them into ~/.debug directory. In switch output mode, these heavy works
block perf from receiving perf events for too long.

Enable no-buildid and no-buildid-cache by default if --switch-output is
provided. Still allow user use --no-no-buildid to explicitly enable
buildid in this case.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461178794-40467-6-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
[ Updated man page ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf record: Force enable --timestamp-filename when --switch-output is provided 2016-04-28T12:58:59+00:00 Wang Nan wangnan0@huawei.com 2016-04-20T18:59:51+00:00 eca857ab381858450ec2f91f5aaae7f2f7a7a180 Without this patch, the last output doesn't have timestamp appended if --timestamp-filename is not explicitly provided. For example: # perf record -a --switch-output & [1] 11224 # kill -s SIGUSR2 11224 [ perf record: dump data: Woken up 1 times ] # [ perf record: Dump perf.data.2015122622372823 ] # fg perf record -a --switch-output ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.027 MB perf.data (540 samples) ] # ls -l total 836 -rw------- 1 root root 33256 Dec 26 22:37 perf.data <---- *Odd* -rw------- 1 root root 817156 Dec 26 22:37 perf.data.2015122622372823 Signed-off-by: Wang Nan <wangnan0@huawei.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Zefan Li <lizefan@huawei.com> Cc: pi3orama@163.com Link: http://lkml.kernel.org/r/1461178794-40467-5-git-send-email-wangnan0@huawei.com Signed-off-by: He Kuang <hekuang@huawei.com> [ Updated man page, that also got an entry for --timestamp-filename ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
Without this patch, the last output doesn't have timestamp appended if
--timestamp-filename is not explicitly provided. For example:

  # perf record -a --switch-output &
  [1] 11224
  # kill -s SIGUSR2 11224
  [ perf record: dump data: Woken up 1 times ]
  # [ perf record: Dump perf.data.2015122622372823 ]

  # fg
  perf record -a --switch-output
  ^C[ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.027 MB perf.data (540 samples) ]

  # ls -l
  total 836
  -rw------- 1 root root  33256 Dec 26 22:37 perf.data   <---- *Odd*
  -rw------- 1 root root 817156 Dec 26 22:37 perf.data.2015122622372823

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461178794-40467-5-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
[ Updated man page, that also got an entry for --timestamp-filename ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf record: Split output into multiple files via '--switch-output' 2016-04-28T12:58:59+00:00 Wang Nan wangnan0@huawei.com 2016-04-20T18:59:50+00:00 3c1cb7e3723caad9b4c1b2f816d86d8605296a4b Allow 'perf record' to split its output into multiple files. For example: # ~/perf record -a --timestamp-filename --switch-output & [1] 10763 # kill -s SIGUSR2 10763 [ perf record: dump data: Woken up 1 times ] # [ perf record: Dump perf.data.2015122622314468 ] # kill -s SIGUSR2 10763 [ perf record: dump data: Woken up 1 times ] # [ perf record: Dump perf.data.2015122622314762 ] # kill -s SIGUSR2 10763 [ perf record: dump data: Woken up 1 times ] #[ perf record: Dump perf.data.2015122622315171 ] # fg perf record -a --timestamp-filename --switch-output ^C[ perf record: Woken up 1 times to write data ] [ perf record: Dump perf.data.2015122622315513 ] [ perf record: Captured and wrote 0.014 MB perf.data.<timestamp> (296 samples) ] # ls -l total 920 -rw------- 1 root root 797692 Dec 26 22:31 perf.data.2015122622314468 -rw------- 1 root root 59960 Dec 26 22:31 perf.data.2015122622314762 -rw------- 1 root root 59912 Dec 26 22:31 perf.data.2015122622315171 -rw------- 1 root root 19220 Dec 26 22:31 perf.data.2015122622315513 Signed-off-by: Wang Nan <wangnan0@huawei.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Zefan Li <lizefan@huawei.com> Cc: pi3orama@163.com Link: http://lkml.kernel.org/r/1461178794-40467-4-git-send-email-wangnan0@huawei.com Signed-off-by: He Kuang <hekuang@huawei.com> [ Added man page entry, used the re-synthesize patch in this series as a fixup ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
Allow 'perf record' to split its output into multiple files.

For example:

  # ~/perf record -a --timestamp-filename --switch-output &
  [1] 10763
  # kill -s SIGUSR2 10763
  [ perf record: dump data: Woken up 1 times ]
  # [ perf record: Dump perf.data.2015122622314468 ]

  # kill -s SIGUSR2 10763
  [ perf record: dump data: Woken up 1 times ]
  # [ perf record: Dump perf.data.2015122622314762 ]

  # kill -s SIGUSR2 10763
  [ perf record: dump data: Woken up 1 times ]
  #[ perf record: Dump perf.data.2015122622315171 ]

  # fg
  perf record -a --timestamp-filename --switch-output
  ^C[ perf record: Woken up 1 times to write data ]
  [ perf record: Dump perf.data.2015122622315513 ]
  [ perf record: Captured and wrote 0.014 MB perf.data.<timestamp> (296 samples) ]

  # ls -l
  total 920
  -rw------- 1 root root 797692 Dec 26 22:31 perf.data.2015122622314468
  -rw------- 1 root root  59960 Dec 26 22:31 perf.data.2015122622314762
  -rw------- 1 root root  59912 Dec 26 22:31 perf.data.2015122622315171
  -rw------- 1 root root  19220 Dec 26 22:31 perf.data.2015122622315513

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461178794-40467-4-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
[ Added man page entry, used the re-synthesize patch in this series as a fixup ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf tools: Set the maximum allowed stack from /proc/sys/kernel/perf_event_max_stack 2016-04-27T13:29:07+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-04-27T13:16:24+00:00 4cb93446c587d56e2a54f4f83113daba2c0b6dee There is an upper limit to what tooling considers a valid callchain, and it was tied to the hardcoded value in the kernel, PERF_MAX_STACK_DEPTH (127), now that this can be tuned via a sysctl, make it read it and use that as the upper limit, falling back to PERF_MAX_STACK_DEPTH for kernels where this sysctl isn't present. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Brendan Gregg <brendan.d.gregg@gmail.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-yjqsd30nnkogvj5oyx9ghir9@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
There is an upper limit to what tooling considers a valid callchain,
and it was tied to the hardcoded value in the kernel,
PERF_MAX_STACK_DEPTH (127), now that this can be tuned via a sysctl,
make it read it and use that as the upper limit, falling back to
PERF_MAX_STACK_DEPTH for kernels where this sysctl isn't present.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-yjqsd30nnkogvj5oyx9ghir9@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf trace: Bump --mmap-pages when --call-graph is used by the root user 2016-04-15T20:52:34+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-04-15T20:52:34+00:00 f3e459d16a8493b617ccf2a940330279679e0291 To reduce the chances we'll overflow the mmap buffer, manual fine tuning trumps this. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-wxygbxmp1v9mng1ea28wet02@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
To reduce the chances we'll overflow the mmap buffer, manual fine tuning
trumps this.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-wxygbxmp1v9mng1ea28wet02@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf trace: Make --(min,max}-stack imply "--call-graph dwarf" 2016-04-15T19:41:19+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-04-15T19:41:19+00:00 056149932602ef905f1e26fc4fe242ef0533a597 If one uses: # perf trace --min-stack 16 Then it implicitly means that callgraphs should be enabled, and the best option in terms of widespread availability is "dwarf". Further work needed to choose a better alternative, LBR, in capable systems. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-xtjmnpkyk42npekxz3kynzmx@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
If one uses:

  # perf trace --min-stack 16

Then it implicitly means that callgraphs should be enabled, and the best
option in terms of widespread availability is "dwarf".

Further work needed to choose a better alternative, LBR, in capable
systems.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-xtjmnpkyk42npekxz3kynzmx@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf trace: Introduce --min-stack filter 2016-04-15T16:14:20+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-04-15T14:10:31+00:00 5cf9c84e21067ec7a44648aedbc38c197d707258 Counterpart to --max-stack, to help focusing on deeply nested calls. Can be combined with --duration, etc. E.g.: System wide syscall tracing looking for call stacks longer than 66: # trace --mmap-pages 32768 --filter-pid 2711 --call-graph dwarf,16384 --min-stack 66 Or more compactly: # trace -m 32768 --filt 2711 --call dwarf,16384 --min-st 66 363.027 ( 0.002 ms): gnome-shell/2287 poll(ufds: 0x7ffc5ea24230, nfds: 1, timeout_msecs: 4294967295 ) = 1 [0xf6fdd] (/usr/lib64/libc-2.22.so) _xcb_conn_wait+0x92 (/usr/lib64/libxcb.so.1.1.0) _xcb_out_send+0x4d (/usr/lib64/libxcb.so.1.1.0) xcb_writev+0x45 (/usr/lib64/libxcb.so.1.1.0) _XSend+0x19e (/usr/lib64/libX11.so.6.3.0) _XReply+0x82 (/usr/lib64/libX11.so.6.3.0) XSync+0x4d (/usr/lib64/libX11.so.6.3.0) dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0) _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1) _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1) _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1) cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1) _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1) cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1) paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0) meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2) g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2) meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0) main+0x3f7 (/usr/bin/gnome-shell) __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so) [0x2909] (/usr/bin/gnome-shell) 363.038 ( 0.006 ms): gnome-shell/2287 writev(fd: 5<socket:[32540]>, vec: 0x7ffc5ea243a0, vlen: 3 ) = 4 __GI___writev+0x2d (/usr/lib64/libc-2.22.so) _xcb_conn_wait+0x359 (/usr/lib64/libxcb.so.1.1.0) _xcb_out_send+0x4d (/usr/lib64/libxcb.so.1.1.0) xcb_writev+0x45 (/usr/lib64/libxcb.so.1.1.0) _XSend+0x19e (/usr/lib64/libX11.so.6.3.0) _XReply+0x82 (/usr/lib64/libX11.so.6.3.0) XSync+0x4d (/usr/lib64/libX11.so.6.3.0) dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0) _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1) _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1) _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1) cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1) _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1) cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1) paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0) meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2) g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2) meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0) main+0x3f7 (/usr/bin/gnome-shell) __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so) [0x2909] (/usr/bin/gnome-shell) 363.086 ( 0.042 ms): gnome-shell/2287 poll(ufds: 0x7ffc5ea24250, nfds: 1, timeout_msecs: 4294967295 ) = 1 [0xf6fdd] (/usr/lib64/libc-2.22.so) _xcb_conn_wait+0x92 (/usr/lib64/libxcb.so.1.1.0) wait_for_reply+0xb7 (/usr/lib64/libxcb.so.1.1.0) xcb_wait_for_reply+0x61 (/usr/lib64/libxcb.so.1.1.0) _XReply+0x127 (/usr/lib64/libX11.so.6.3.0) XSync+0x4d (/usr/lib64/libX11.so.6.3.0) dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0) _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1) _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1) _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1) _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1) cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1) _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1) cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1) paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0) meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so) _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2) meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0) _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2) g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2) clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2) _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2) clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2) g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2) g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2) meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0) main+0x3f7 (/usr/bin/gnome-shell) __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so) [0x2909] (/usr/bin/gnome-shell) Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-jncuxju9fibq2rl6olhqwjw6@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
Counterpart to --max-stack, to help focusing on deeply nested calls. Can
be combined with --duration, etc.

E.g.:

  System wide syscall tracing looking for call stacks longer than 66:

  # trace --mmap-pages 32768 --filter-pid 2711 --call-graph dwarf,16384 --min-stack 66

  Or more compactly:

  # trace -m 32768 --filt 2711 --call dwarf,16384 --min-st 66
   363.027 ( 0.002 ms): gnome-shell/2287 poll(ufds: 0x7ffc5ea24230, nfds: 1, timeout_msecs: 4294967295         ) = 1
                                       [0xf6fdd] (/usr/lib64/libc-2.22.so)
                                       _xcb_conn_wait+0x92 (/usr/lib64/libxcb.so.1.1.0)
                                       _xcb_out_send+0x4d (/usr/lib64/libxcb.so.1.1.0)
                                       xcb_writev+0x45 (/usr/lib64/libxcb.so.1.1.0)
                                       _XSend+0x19e (/usr/lib64/libX11.so.6.3.0)
                                       _XReply+0x82 (/usr/lib64/libX11.so.6.3.0)
                                       XSync+0x4d (/usr/lib64/libX11.so.6.3.0)
                                       dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0)
                                       _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1)
                                       paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0)
                                       meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0)
                                       main+0x3f7 (/usr/bin/gnome-shell)
                                       __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so)
                                       [0x2909] (/usr/bin/gnome-shell)
   363.038 ( 0.006 ms): gnome-shell/2287 writev(fd: 5<socket:[32540]>, vec: 0x7ffc5ea243a0, vlen: 3            ) = 4
                                       __GI___writev+0x2d (/usr/lib64/libc-2.22.so)
                                       _xcb_conn_wait+0x359 (/usr/lib64/libxcb.so.1.1.0)
                                       _xcb_out_send+0x4d (/usr/lib64/libxcb.so.1.1.0)
                                       xcb_writev+0x45 (/usr/lib64/libxcb.so.1.1.0)
                                       _XSend+0x19e (/usr/lib64/libX11.so.6.3.0)
                                       _XReply+0x82 (/usr/lib64/libX11.so.6.3.0)
                                       XSync+0x4d (/usr/lib64/libX11.so.6.3.0)
                                       dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0)
                                       _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1)
                                       paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0)
                                       meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0)
                                       main+0x3f7 (/usr/bin/gnome-shell)
                                       __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so)
                                       [0x2909] (/usr/bin/gnome-shell)
   363.086 ( 0.042 ms): gnome-shell/2287 poll(ufds: 0x7ffc5ea24250, nfds: 1, timeout_msecs: 4294967295         ) = 1
                                       [0xf6fdd] (/usr/lib64/libc-2.22.so)
                                       _xcb_conn_wait+0x92 (/usr/lib64/libxcb.so.1.1.0)
                                       wait_for_reply+0xb7 (/usr/lib64/libxcb.so.1.1.0)
                                       xcb_wait_for_reply+0x61 (/usr/lib64/libxcb.so.1.1.0)
                                       _XReply+0x127 (/usr/lib64/libX11.so.6.3.0)
                                       XSync+0x4d (/usr/lib64/libX11.so.6.3.0)
                                       dri3_bind_tex_image+0x42 (/usr/lib64/libGL.so.1.2.0)
                                       _cogl_winsys_texture_pixmap_x11_update+0x117 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_update+0x67 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_texture_pixmap_x11_pre_paint+0x13 (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_pipeline_layer_pre_paint+0x5e (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_rectangles_validate_layer_cb+0x1b (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_pipeline_foreach_layer+0xbe (/usr/lib64/libcogl.so.20.4.1)
                                       _cogl_framebuffer_draw_multitextured_rectangles+0x77 (/usr/lib64/libcogl.so.20.4.1)
                                       cogl_framebuffer_draw_multitextured_rectangle+0x51 (/usr/lib64/libcogl.so.20.4.1)
                                       paint_clipped_rectangle+0xb6 (/usr/lib64/libmutter.so.0.0.0)
                                       meta_shaped_texture_paint+0x3e3 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_actor_paint+0x14b (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_real_paint+0x20 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_window_group_paint+0x19f (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       [0x3d970] (/usr/lib64/gnome-shell/libgnome-shell.so)
                                       _g_closure_invoke_va+0xb2 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_paint+0x3a (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       meta_stage_paint+0x45 (/usr/lib64/libmutter.so.0.0.0)
                                       _g_closure_invoke_va+0x164 (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit_valist+0xc0d (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       g_signal_emit+0x8f (/usr/lib64/libgobject-2.0.so.0.4600.2)
                                       clutter_actor_continue_paint+0x2bb (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_actor_paint.part.41+0x47b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_paint+0x17b (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_stage_cogl_redraw+0x496 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       _clutter_stage_do_update+0x117 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       clutter_clock_dispatch+0x169 (/usr/lib64/libclutter-1.0.so.0.2400.2)
                                       g_main_context_dispatch+0x15a (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_context_iterate.isra.29+0x1e0 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       g_main_loop_run+0xc2 (/usr/lib64/libglib-2.0.so.0.4600.2)
                                       meta_run+0x2c (/usr/lib64/libmutter.so.0.0.0)
                                       main+0x3f7 (/usr/bin/gnome-shell)
                                       __libc_start_main+0xf0 (/usr/lib64/libc-2.22.so)
                                       [0x2909] (/usr/bin/gnome-shell)

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-jncuxju9fibq2rl6olhqwjw6@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf trace: Add --max-stack knob 2016-04-14T22:46:58+00:00 Arnaldo Carvalho de Melo acme@redhat.com 2016-04-14T21:29:08+00:00 c6d4a494a207a336b45e52a44550150964daf1ce Similar to the one in the other tools (report, script, top). Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Milian Wolff <milian.wolff@kdab.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: http://lkml.kernel.org/n/tip-lh7kk5a5t3erwxw31ah0cgar@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> 
Similar to the one in the other tools (report, script, top).

Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-lh7kk5a5t3erwxw31ah0cgar@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>