# trace-cmd record -e sched_wakeup -T -F sleep 1 CPU0 data recorded at offset=0xff000 0 bytes in size CPU1 data recorded at offset=0xff000 4096 bytes in size # trace-cmd report unknown option 9 CPU 0 is empty cpus=2 <idle>-0 [001] 2339.917223: sched_wakeup: comm=sleep pid=914 prio=120 target_cpu=001 <idle>-0 [001] 2339.917237: kernel_stack: => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s sleep-914 [001] 2339.917463: sched_wakeup: comm=trace-cmd pid=911 prio=120 target_cpu=001 sleep-914 [001] 2339.917471: kernel_stack: => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s => (nil)s
What kernel was this? That can happen if there's an issue with the kernel stack tracing. What do you see if you do: # trace-cmd start -e sched_wakeup # trace-cmd show If you see something different, then it's trace-cmd that's the problem, but if you still see the "nils" (or zeros) then its coming from the kernel. Thanks! -- Steve
Created attachment 286037 [details] .config (In reply to Steven Rostedt from comment #1) > What kernel was this? That can happen if there's an issue with the kernel > stack tracing. Linux 5.4-rc8. .config attached. > What do you see if you do: > > # trace-cmd start -e sched_wakeup > # trace-cmd show > > If you see something different, then it's trace-cmd that's the problem, but > if you still see the "nils" (or zeros) then its coming from the kernel. # trace-cmd start -e sched_wakeup -T # trace-cmd show # tracer: nop # # entries-in-buffer/entries-written: 144/144 #P:2 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | <idle>-0 [001] dNh. 244.775779: sched_wakeup: comm=trace-cmd pid=876 prio=120 target_cpu=001 <idle>-0 [001] dNh. 244.775795: <stack trace> => trace_event_raw_event_sched_wakeup_template => ttwu_do_wakeup.isra.0 => sched_ttwu_pending => scheduler_ipi => reschedule_interrupt => default_idle => do_idle => cpu_startup_entry => start_secondary => secondary_startup_64 <idle>-0 [001] dNh. 244.775954: sched_wakeup: comm=trace-cmd pid=876 prio=120 target_cpu=001 <idle>-0 [001] dNh. 244.775959: <stack trace> => trace_event_raw_event_sched_wakeup_template => ttwu_do_wakeup.isra.0 => sched_ttwu_pending => scheduler_ipi => reschedule_interrupt => default_idle => do_idle => cpu_startup_entry => start_secondary => secondary_startup_64 (snip)
The result of bisect: b2cf2e46a6d013d4037118274cc783570277ce32 is the first bad commit commit b2cf2e46a6d013d4037118274cc783570277ce32 Author: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com> Date: Fri Aug 2 14:00:58 2019 +0300 trace-cmd: Remove trace-cmd plugin handling routines Currently there are no trace-cmd related plugins, all of them are designed to be used with libtraceeevnt. As both libtraceevent and trace-cmd have logic for managing plugins, the one in trace-cmd is redundant. Those redundant code is removed and replaced with calls to libtraceeevnt plugin APIs. When trace-cmd has to load any plugins, it uses libtraceeevnt to do the job. Removed trace-cmd functions: tracecmd_load_plugins() tracecmd_unload_plugins() trace_util_load_plugins() trace_util_read_plugin_options() trace_util_free_options() trace_util_print_plugins() trace_util_free_plugin_options_list() A new libtraceevent API is added: tep_load_plugins_hook() - the local static function load_plugins() is exposed as API, as this functionality is needed be trace-cmd. Link: http://lore.kernel.org/linux-trace-devel/20190802110101.14759-4-tz.stoyanov@gmail.com Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> include/trace-cmd/trace-cmd.h | 18 --- include/traceevent/event-parse.h | 6 + lib/trace-cmd/trace-input.c | 9 +- lib/trace-cmd/trace-util.c | 330 +-------------------------------------- lib/traceevent/event-plugin.c | 19 ++- plugins/plugin_python.c | 9 +- tracecmd/trace-check-events.c | 10 +- tracecmd/trace-list.c | 21 ++- 8 files changed, 54 insertions(+), 368 deletions(-)
Hi Kusanagi, The problem seems to be in the tracevent's function plugin,it was not loaded by trace-cmd report. Please, can you check what plugins are loaded, by executing "trace-cmd list -P" ? Depending on how trace-cmd is compiled and installed, plugins should be in /usr/local/lib/traceevent/plugins/ or /usr/lib/traceevent/plugins/. Please, can you also search for file named "plugin_function.so", this is the missing plugin. Thank You very much for reporting and analysing the problem ! Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
(In reply to Tzvetomir Stoyanov from comment #4) I run trace-cmd from a build dir without installing. That is the cause. If I set TRACEEVENT_PLUGIN_DIR, stacktrace works as expexted. I'll resolve this bug as INVALID. Thanks.