Bug 204231
| Summary: | Python 2 only functions being used, non-op with Python 3 | ||
|---|---|---|---|
| Product: | Tools | Reporter: | Troy Engel (tengel) |
| Component: | Trace-cmd/Kernelshark | Assignee: | Default virtual assignee for Trace-cmd and kernelshark (tools_tracecmd_kernelshark) |
| Status: | RESOLVED CODE_FIX | ||
| Severity: | low | CC: | howaboutsynergy, rostedt |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 5.2.0-arch2-1-ARCH | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
Add support for Python 3
Add support for Python 3 v2 |
||
Created attachment 283841 [details]
Add support for Python 3
Hi Troy,
Does the attached patch fix it for you. Note, I was not the one that added this code. I just did some googling to see Python 3 replacements for the functions that appeared to have issues.
Created attachment 283843 [details]
Add support for Python 3 v2
I had some extra changes in that patch. This is the patch that I plan on adding (it will be three patches).
> Add support for Python 3 v2 tl;dr: patch seems to work, here's how I tested it: I replaced all 4 `python2` occurrences in latest PKGBUILD https://aur.archlinux.org/cgit/aur.git/commit/?h=trace-cmd-git&id=d0913fea414ed795fdcc1c5c86a00fb030ec024d with `python`, then applied patch from Comment 2 and this one(fixes typo `cound` => `could`): https://github.com/howaboutsynergy/q1q/blob/f4bd9047a8fefa38db774f8087ecd06c94f4dc78/OSes/archlinux/home/user/build/1packages/4used/trace-cmd/typo.patch#L1 recompiled kernel with tracing on (so that this would work `$ sudo trace-cmd record -e compaction:* sleep 10`): ``` $ diffkconfigs .config.lastgoodNOTRACE .config Comparing: --- '.config.lastgoodNOTRACE' +++ '.config' --- /dev/fd/63 2019-07-19 23:50:47.046989084 +0200 +++ /dev/fd/62 2019-07-19 23:50:47.046989084 +0200 +CONFIG_BINARY_PRINTF=y +CONFIG_BLK_DEV_IO_TRACE=y +CONFIG_BPF_EVENTS=y +CONFIG_BRANCH_TRACER=y +CONFIG_CONTEXT_SWITCH_TRACER=y +CONFIG_DYNAMIC_EVENTS=y +CONFIG_DYNAMIC_FTRACE_WITH_REGS=y +CONFIG_DYNAMIC_FTRACE=y +CONFIG_EVENT_TRACING=y +CONFIG_FTRACE_MCOUNT_RECORD=y +CONFIG_FTRACE_SYSCALLS=y +CONFIG_FTRACE=y +CONFIG_FUNCTION_GRAPH_TRACER=y +CONFIG_FUNCTION_PROFILER=y +CONFIG_FUNCTION_TRACER=y +CONFIG_GENERIC_TRACER=y +CONFIG_HIST_TRIGGERS=y +CONFIG_HWLAT_TRACER=y +CONFIG_IRQSOFF_TRACER=y +CONFIG_MMIOTRACE=y +CONFIG_NOP_TRACER=y +CONFIG_PERCPU_STATS=y +CONFIG_PREEMPTIRQ_EVENTS=y +CONFIG_PREEMPTIRQ_TRACEPOINTS=y +CONFIG_PROBE_EVENTS=y +CONFIG_PROFILE_ANNOTATED_BRANCHES=y +CONFIG_RING_BUFFER_ALLOW_SWAP=y +CONFIG_RING_BUFFER_STARTUP_TEST=y +CONFIG_RING_BUFFER=y +CONFIG_SCHED_TRACER=y +CONFIG_STACK_TRACER=y +CONFIG_TRACE_BRANCH_PROFILING=y +CONFIG_TRACE_CLOCK=y +CONFIG_TRACE_EVAL_MAP_FILE=y +CONFIG_TRACE_IRQFLAGS=y +CONFIG_TRACEPOINTS=y +CONFIG_TRACER_MAX_TRACE=y +CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y +CONFIG_TRACER_SNAPSHOT=y +CONFIG_TRACING_BRANCHES=y +CONFIG_TRACING_MAP=y +CONFIG_TRACING=y +CONFIG_UPROBE_EVENTS=y +CONFIG_UPROBES=y ``` rebooted into new kernel then `$ cd /tmp` (started some craziness on another terminal: `$ while true; do time stress -m 220 --vm-bytes 10000000000 --timeout 10; done`) ``` $ sudo trace-cmd record -e compaction:* sleep 20 [sudo] password for user: CPU0 data recorded at offset=0x5bc000 311296 bytes in size CPU1 data recorded at offset=0x608000 299008 bytes in size CPU2 data recorded at offset=0x651000 335872 bytes in size CPU3 data recorded at offset=0x6a3000 544768 bytes in size CPU4 data recorded at offset=0x728000 286720 bytes in size CPU5 data recorded at offset=0x76e000 417792 bytes in size CPU6 data recorded at offset=0x7d4000 188416 bytes in size CPU7 data recorded at offset=0x802000 253952 bytes in size CPU8 data recorded at offset=0x840000 167936 bytes in size CPU9 data recorded at offset=0x869000 327680 bytes in size CPU10 data recorded at offset=0x8b9000 618496 bytes in size CPU11 data recorded at offset=0x950000 516096 bytes in size $ sudo trace-cmd report > trace.log $ l trace* -rw-r--r-- 1 root root 10280960 19.07.2019 23:54 trace.dat -rw-r--r-- 1 user user 15610338 19.07.2019 23:55 trace.log ``` As you can see, no errors were outputted(also no 'py' in trace.log, case insensitive search - I went overboard just in case). So patch must've worked for my particular test case here, unless I'm missing something(hopefully it did use python 3 instead of 2). Previously(w/o this patch, that is) the error(s) would be like this: https://aur.archlinux.org/packages/trace-cmd-git/#comment-700805 Thanks for this patch, Steven. I'll keep using it for now. PS: kernel used 5.2.1-g527a3db363a3 with compaction patch from here: https://bugzilla.kernel.org/show_bug.cgi?id=204165#c24 otherwise trace.log could've been like 4GiB. Thanks for verifying. And I just released 2.8.2, where I should have waited. Oh well, now I'll add a 2.8.3 ;-) Hi Steven - howaboutsynergy is said user who reported this of course, their positive results are deeper/better than I could have tested. :) Ship it! for completion/correction purposes, and this is a different issue, but I've noticed something that slipped my test(s): if trace-cmd-git isn't already installed on my system(ArchLinux), then compilation will fail. I used to workaround this before by not using `gui` and `install_gui` in the PKGBUILD's `make` invocations. (already filed: https://bugzilla.kernel.org/show_bug.cgi?id=204259 for this) to give you an idea: 1. first remove trace-cmd from system: `$ sudo pacman -Rs trace-cmd-git ; sudo pacman -Rs trace-cmd` 2. try to compile it, using latest PKGBUILD https://aur.archlinux.org/cgit/aur.git/commit/?h=trace-cmd-git&id=d0913fea414ed795fdcc1c5c86a00fb030ec024d ``` ... -- Detecting CXX compile features - done project: Kernel Shark: (version: 0.9.8) CMake Error at build/FindTraceCmd.cmake:58 (MESSAGE): Could not find trace-cmd! Call Stack (most recent call first): CMakeLists.txt:16 (include) -- Configuring incomplete, errors occurred! See also "/home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/CMakeFiles/CMakeOutput.log". See also "/home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/CMakeFiles/CMakeError.log". make: *** [Makefile:259: /home/user/build/1packages/4used/trace-cmd/makepkg_pacman/trace-cmd-git/src/trace-cmd/kernel-shark/build/Makefile] Error 1 make: *** Waiting for unfinished jobs.... ASCIIDOC trace-cmd-start.1.xsl ``` @howaboutsynergy - yes, this is a current artifact of our current trace-cmd-git packaging and testing, the very latest git changes are now searching the path for the `trace-cmd` binary instead of the previous methods just awhile back (where it would use the local copy just compiled). It's basically this commit right here which I have not caught up with in the PKGBUILD yet: https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/commit/kernel-shark/build/FindTraceCmd.cmake?id=1b25ee0b18702ae8a6afff2db234bceab52bdd3c We're at the tail end of KS 1.0 dev, now that it's basically ready for a release this is going to become two packages in Arch (`trace-cmd` and `kernelshark`) linked with standard dependencies. Steven sent an email a few days ago, he's hoping to tag KS 1.0 this week which will be our signal to move forward and take the next steps in splitting up the packaging. Fixed by commits:
ff3c2189ddbd ("trace-cmd: Replace PySting_FromString() with PyUnicode_FromString()")
dbddc64834a7 ("trace-cmd: Use PyMemoryView_FromMemory() for Python 3")
2091de2221e9 ("trace-cmd: Use PyLong_AsLong() for Python 3")
Thanks. I'm kinda lazy to make another bug for the "`cound`" typo mentioned in Comment 3 Any chance someone could just apply it anyway? I'm only asking because it doesn't seem present in master already: https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/log/ trace-cmd-git 2.7.r457.g2091de2-1 |
Hi team, latest git commit as of 2019-07-19 08:45 CDT A user of the Arch package reported this error in general usage when the package is compiled using Python 3: $ trace-cmd report > trace.log cound not load plugin '/usr/lib/trace-cmd/plugins/plugin_python.so' /usr/lib/trace-cmd/plugins/plugin_python.so: undefined symbol: PyString_FromString This is found in the python/ctracecmd.i file, we're building the package using these options: make BUILD_TYPE=Release PYTHON_VERS=python3 \ prefix="/usr" DESTDIR="$pkgdir" all doc gui A Google tells me `PyString_FromString` is py2 only and needs replaced (PyUnicode_FromString?) but I imagine there are others, this is just the one that popped out of the woodwork for the user. I'm not a programmer by trade. :) I've reverted our package to name PYTHON_VERS=python2 and it's functional as expected. Python 2 goes EOL/fully deprecated on 1 Jan 2020 (less than 6 months). Thanks!