Bug 205643
Summary: | Stacktrace is all "(nil)s" | ||
---|---|---|---|
Product: | Tools | Reporter: | Kusanagi Kouichi (slash) |
Component: | Trace-cmd/Kernelshark | Assignee: | Default virtual assignee for Trace-cmd and kernelshark (tools_tracecmd_kernelshark) |
Status: | RESOLVED INVALID | ||
Severity: | normal | CC: | rostedt, tstoyanov |
Priority: | P1 | ||
Hardware: | x86-64 | ||
OS: | Linux | ||
Kernel Version: | 5.4-rc8 | Subsystem: | |
Regression: | No | Bisected commit-id: | |
Attachments: | .config |
Description
Kusanagi Kouichi
2019-11-24 12:32:26 UTC
What kernel was this? That can happen if there's an issue with the kernel stack tracing. What do you see if you do: # trace-cmd start -e sched_wakeup # trace-cmd show If you see something different, then it's trace-cmd that's the problem, but if you still see the "nils" (or zeros) then its coming from the kernel. Thanks! -- Steve Created attachment 286037 [details] .config (In reply to Steven Rostedt from comment #1) > What kernel was this? That can happen if there's an issue with the kernel > stack tracing. Linux 5.4-rc8. .config attached. > What do you see if you do: > > # trace-cmd start -e sched_wakeup > # trace-cmd show > > If you see something different, then it's trace-cmd that's the problem, but > if you still see the "nils" (or zeros) then its coming from the kernel. # trace-cmd start -e sched_wakeup -T # trace-cmd show # tracer: nop # # entries-in-buffer/entries-written: 144/144 #P:2 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | <idle>-0 [001] dNh. 244.775779: sched_wakeup: comm=trace-cmd pid=876 prio=120 target_cpu=001 <idle>-0 [001] dNh. 244.775795: <stack trace> => trace_event_raw_event_sched_wakeup_template => ttwu_do_wakeup.isra.0 => sched_ttwu_pending => scheduler_ipi => reschedule_interrupt => default_idle => do_idle => cpu_startup_entry => start_secondary => secondary_startup_64 <idle>-0 [001] dNh. 244.775954: sched_wakeup: comm=trace-cmd pid=876 prio=120 target_cpu=001 <idle>-0 [001] dNh. 244.775959: <stack trace> => trace_event_raw_event_sched_wakeup_template => ttwu_do_wakeup.isra.0 => sched_ttwu_pending => scheduler_ipi => reschedule_interrupt => default_idle => do_idle => cpu_startup_entry => start_secondary => secondary_startup_64 (snip) The result of bisect: b2cf2e46a6d013d4037118274cc783570277ce32 is the first bad commit commit b2cf2e46a6d013d4037118274cc783570277ce32 Author: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com> Date: Fri Aug 2 14:00:58 2019 +0300 trace-cmd: Remove trace-cmd plugin handling routines Currently there are no trace-cmd related plugins, all of them are designed to be used with libtraceeevnt. As both libtraceevent and trace-cmd have logic for managing plugins, the one in trace-cmd is redundant. Those redundant code is removed and replaced with calls to libtraceeevnt plugin APIs. When trace-cmd has to load any plugins, it uses libtraceeevnt to do the job. Removed trace-cmd functions: tracecmd_load_plugins() tracecmd_unload_plugins() trace_util_load_plugins() trace_util_read_plugin_options() trace_util_free_options() trace_util_print_plugins() trace_util_free_plugin_options_list() A new libtraceevent API is added: tep_load_plugins_hook() - the local static function load_plugins() is exposed as API, as this functionality is needed be trace-cmd. Link: http://lore.kernel.org/linux-trace-devel/20190802110101.14759-4-tz.stoyanov@gmail.com Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> include/trace-cmd/trace-cmd.h | 18 --- include/traceevent/event-parse.h | 6 + lib/trace-cmd/trace-input.c | 9 +- lib/trace-cmd/trace-util.c | 330 +-------------------------------------- lib/traceevent/event-plugin.c | 19 ++- plugins/plugin_python.c | 9 +- tracecmd/trace-check-events.c | 10 +- tracecmd/trace-list.c | 21 ++- 8 files changed, 54 insertions(+), 368 deletions(-) Hi Kusanagi, The problem seems to be in the tracevent's function plugin,it was not loaded by trace-cmd report. Please, can you check what plugins are loaded, by executing "trace-cmd list -P" ? Depending on how trace-cmd is compiled and installed, plugins should be in /usr/local/lib/traceevent/plugins/ or /usr/lib/traceevent/plugins/. Please, can you also search for file named "plugin_function.so", this is the missing plugin. Thank You very much for reporting and analysing the problem ! Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center (In reply to Tzvetomir Stoyanov from comment #4) I run trace-cmd from a build dir without installing. That is the cause. If I set TRACEEVENT_PLUGIN_DIR, stacktrace works as expexted. I'll resolve this bug as INVALID. Thanks. |