Bug 204231 - Python 2 only functions being used, non-op with Python 3
Summary: Python 2 only functions being used, non-op with Python 3
Status: RESOLVED CODE_FIX
Alias: None
Product: Tools
Classification: Unclassified
Component: Trace-cmd/Kernelshark (show other bugs)
Hardware: All Linux
: P1 low
Assignee: Default virtual assignee for Trace-cmd and kernelshark
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-19 13:50 UTC by Troy Engel
Modified: 2019-07-23 14:17 UTC (History)
2 users (show)

See Also:
Kernel Version: 5.2.0-arch2-1-ARCH
Tree: Mainline
Regression: No


Attachments
Add support for Python 3 (2.88 KB, patch)
2019-07-19 18:40 UTC, Steven Rostedt
Details | Diff
Add support for Python 3 v2 (2.44 KB, patch)
2019-07-19 18:43 UTC, Steven Rostedt
Details | Diff

Description Troy Engel 2019-07-19 13:50:32 UTC
Hi team, latest git commit as of 2019-07-19 08:45 CDT

A user of the Arch package reported this error in general usage when the package is compiled using Python 3:

  $ trace-cmd report > trace.log  
    cound not load plugin '/usr/lib/trace-cmd/plugins/plugin_python.so'
/usr/lib/trace-cmd/plugins/plugin_python.so: undefined symbol: PyString_FromString

This is found in the python/ctracecmd.i file, we're building the package using these options:

  make BUILD_TYPE=Release PYTHON_VERS=python3 \
    prefix="/usr" DESTDIR="$pkgdir" all doc gui

A Google tells me `PyString_FromString` is py2 only and needs replaced (PyUnicode_FromString?) but I imagine there are others, this is just the one that popped out of the woodwork for the user. I'm not a programmer by trade. :)

I've reverted our package to name PYTHON_VERS=python2 and it's functional as expected. Python 2 goes EOL/fully deprecated on 1 Jan 2020 (less than 6 months). Thanks!
Comment 1 Steven Rostedt 2019-07-19 18:40:40 UTC
Created attachment 283841 [details]
Add support for Python 3

Hi Troy,

Does the attached patch fix it for you. Note, I was not the one that added this code. I just did some googling to see Python 3 replacements for the functions that appeared to have issues.
Comment 2 Steven Rostedt 2019-07-19 18:43:48 UTC
Created attachment 283843 [details]
Add support for Python 3 v2

I had some extra changes in that patch. This is the patch that I plan on adding (it will be three patches).
Comment 3 GYt2bW 2019-07-19 22:06:39 UTC
> Add support for Python 3 v2

tl;dr: patch seems to work, here's how I tested it:

I replaced all 4 `python2` occurrences in latest PKGBUILD 
https://aur.archlinux.org/cgit/aur.git/commit/?h=trace-cmd-git&id=d0913fea414ed795fdcc1c5c86a00fb030ec024d
with `python`, then applied patch from Comment 2
and this one(fixes typo `cound` => `could`):
https://github.com/howaboutsynergy/q1q/blob/f4bd9047a8fefa38db774f8087ecd06c94f4dc78/OSes/archlinux/home/user/build/1packages/4used/trace-cmd/typo.patch#L1

recompiled kernel with tracing on (so that this would work `$ sudo trace-cmd record -e compaction:* sleep 10`):
```
$ diffkconfigs .config.lastgoodNOTRACE  .config
Comparing:
--- '.config.lastgoodNOTRACE'
+++ '.config'
--- /dev/fd/63	2019-07-19 23:50:47.046989084 +0200
+++ /dev/fd/62	2019-07-19 23:50:47.046989084 +0200
+CONFIG_BINARY_PRINTF=y
+CONFIG_BLK_DEV_IO_TRACE=y
+CONFIG_BPF_EVENTS=y
+CONFIG_BRANCH_TRACER=y
+CONFIG_CONTEXT_SWITCH_TRACER=y
+CONFIG_DYNAMIC_EVENTS=y
+CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
+CONFIG_DYNAMIC_FTRACE=y
+CONFIG_EVENT_TRACING=y
+CONFIG_FTRACE_MCOUNT_RECORD=y
+CONFIG_FTRACE_SYSCALLS=y
+CONFIG_FTRACE=y
+CONFIG_FUNCTION_GRAPH_TRACER=y
+CONFIG_FUNCTION_PROFILER=y
+CONFIG_FUNCTION_TRACER=y
+CONFIG_GENERIC_TRACER=y
+CONFIG_HIST_TRIGGERS=y
+CONFIG_HWLAT_TRACER=y
+CONFIG_IRQSOFF_TRACER=y
+CONFIG_MMIOTRACE=y
+CONFIG_NOP_TRACER=y
+CONFIG_PERCPU_STATS=y
+CONFIG_PREEMPTIRQ_EVENTS=y
+CONFIG_PREEMPTIRQ_TRACEPOINTS=y
+CONFIG_PROBE_EVENTS=y
+CONFIG_PROFILE_ANNOTATED_BRANCHES=y
+CONFIG_RING_BUFFER_ALLOW_SWAP=y
+CONFIG_RING_BUFFER_STARTUP_TEST=y
+CONFIG_RING_BUFFER=y
+CONFIG_SCHED_TRACER=y
+CONFIG_STACK_TRACER=y
+CONFIG_TRACE_BRANCH_PROFILING=y
+CONFIG_TRACE_CLOCK=y
+CONFIG_TRACE_EVAL_MAP_FILE=y
+CONFIG_TRACE_IRQFLAGS=y
+CONFIG_TRACEPOINTS=y
+CONFIG_TRACER_MAX_TRACE=y
+CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y
+CONFIG_TRACER_SNAPSHOT=y
+CONFIG_TRACING_BRANCHES=y
+CONFIG_TRACING_MAP=y
+CONFIG_TRACING=y
+CONFIG_UPROBE_EVENTS=y
+CONFIG_UPROBES=y
```

rebooted into new kernel
then
`$ cd /tmp`
(started some craziness on another terminal: `$ while true; do time stress -m 220 --vm-bytes 10000000000 --timeout 10; done`)
```
$ sudo trace-cmd record -e compaction:* sleep 20
[sudo] password for user: 
CPU0 data recorded at offset=0x5bc000
    311296 bytes in size
CPU1 data recorded at offset=0x608000
    299008 bytes in size
CPU2 data recorded at offset=0x651000
    335872 bytes in size
CPU3 data recorded at offset=0x6a3000
    544768 bytes in size
CPU4 data recorded at offset=0x728000
    286720 bytes in size
CPU5 data recorded at offset=0x76e000
    417792 bytes in size
CPU6 data recorded at offset=0x7d4000
    188416 bytes in size
CPU7 data recorded at offset=0x802000
    253952 bytes in size
CPU8 data recorded at offset=0x840000
    167936 bytes in size
CPU9 data recorded at offset=0x869000
    327680 bytes in size
CPU10 data recorded at offset=0x8b9000
    618496 bytes in size
CPU11 data recorded at offset=0x950000
    516096 bytes in size
$ sudo trace-cmd report > trace.log
$ l trace*
-rw-r--r-- 1 root root 10280960 19.07.2019 23:54 trace.dat
-rw-r--r-- 1 user user 15610338 19.07.2019 23:55 trace.log
```

As you can see, no errors were outputted(also no 'py' in trace.log, case insensitive search - I went overboard just in case). So patch must've worked for my particular test case here, unless I'm missing something(hopefully it did use python 3 instead of 2).
Previously(w/o this patch, that is) the error(s) would be like this: https://aur.archlinux.org/packages/trace-cmd-git/#comment-700805

Thanks for this patch, Steven. I'll keep using it for now.

PS: kernel used 5.2.1-g527a3db363a3 with compaction patch from here: https://bugzilla.kernel.org/show_bug.cgi?id=204165#c24  otherwise trace.log could've been like 4GiB.
Comment 4 Steven Rostedt 2019-07-19 22:34:14 UTC
Thanks for verifying. And I just released 2.8.2, where I should have waited. Oh well, now I'll add a 2.8.3 ;-)
Comment 5 Troy Engel 2019-07-20 12:57:21 UTC
Hi Steven -  howaboutsynergy is said user who reported this of course, their positive results are deeper/better than I could have tested. :) Ship it!
Comment 6 GYt2bW 2019-07-21 23:45:03 UTC
for completion/correction purposes, and this is a different issue, but I've noticed something that slipped my test(s): if trace-cmd-git isn't already installed on my system(ArchLinux), then compilation will fail. I used to workaround this before by not using `gui` and `install_gui` in the PKGBUILD's `make` invocations.

(already filed: https://bugzilla.kernel.org/show_bug.cgi?id=204259 for this)

to give you an idea:

1. first remove trace-cmd from system:
`$ sudo pacman -Rs trace-cmd-git ; sudo pacman -Rs trace-cmd`

2. try to compile it, using latest PKGBUILD https://aur.archlinux.org/cgit/aur.git/commit/?h=trace-cmd-git&id=d0913fea414ed795fdcc1c5c86a00fb030ec024d



```
...
-- Detecting CXX compile features - done

 project: Kernel Shark: (version: 0.9.8)

CMake Error at build/FindTraceCmd.cmake:58 (MESSAGE):
  

  Could not find trace-cmd!

Call Stack (most recent call first):
  CMakeLists.txt:16 (include)


-- Configuring incomplete, errors occurred!
See also "/home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/CMakeFiles/CMakeOutput.log".
See also "/home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:259: /home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/Makefile] Error 1
make: *** Waiting for unfinished jobs....
      ASCIIDOC            trace-cmd-start.1.xsl
```
Comment 7 Troy Engel 2019-07-22 23:45:08 UTC
@howaboutsynergy - yes, this is a current artifact of our current trace-cmd-git packaging and testing, the very latest git changes are now searching the path for the `trace-cmd` binary instead of the previous methods just awhile back (where it would use the local copy just compiled). It's basically this commit right here which I have not caught up with in the PKGBUILD yet: https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/commit/kernel-shark/build/FindTraceCmd.cmake?id=1b25ee0b18702ae8a6afff2db234bceab52bdd3c

We're at the tail end of KS 1.0 dev, now that it's basically ready for a release this is going to become two packages in Arch (`trace-cmd` and `kernelshark`) linked with standard dependencies. Steven sent an email a few days ago, he's hoping to tag KS 1.0 this week which will be our signal to move forward and take the next steps in splitting up the packaging.
Comment 8 Steven Rostedt 2019-07-23 13:57:25 UTC
Fixed by commits:

ff3c2189ddbd ("trace-cmd: Replace PySting_FromString() with PyUnicode_FromString()")
dbddc64834a7 ("trace-cmd: Use PyMemoryView_FromMemory() for Python 3")
2091de2221e9 ("trace-cmd: Use PyLong_AsLong() for Python 3")
Comment 9 GYt2bW 2019-07-23 14:17:57 UTC
Thanks.

I'm kinda lazy to make another bug for the "`cound`" typo mentioned in Comment 3
Any chance someone could just apply it anyway?

I'm only asking because it doesn't seem present in master already: https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/log/

trace-cmd-git 2.7.r457.g2091de2-1

Note You need to log in before you can comment on or make changes to this bug.