Hello. Kvm seems to be broken on AMD Athlon64 4000+ Dual Core 2.1GHz Brisbane. A kernel build takes over 2 hours, compared to 15 minutes on the host. I'm running with virtio for storage, and this does not seem to be related to IO. A simple test with dd also demonstrates that the problem is cpu-ralated: Host test: dd if=/dev/zero of=/dev/null bs=1k count=8M 8388608+0 записей считано (read) 8388608+0 записей написано (write) скопировано (copyed) 8589934592 байта (8,6 GB), 7,04456 c, 1,2 GB/c This also does not seem to be a cpu bug, as both VirtualBox and VMware work fine. Vmware/Virtualbox test: dd if=/dev/zero of=/dev/null bs=1k count=8M 8388608+0 записей считано (read) 8388608+0 записей написано (write) скопировано (copyed) 8589934592 байта (8,6 GB), 5,02745 c, 1,7 GB/c I've tried different kernels on host, between 2.6.22 and current git, with the same results. The guest is running Gentoo Hardened kernel 3.2.2 , but I could try with a an other distro and a vanilla kernel if required. My kvm version is kvm 1.0 Help would be much appreciated.
in kvm guest this speed are about 2MB/s and is slow like a hell. With it booting for about 15 minutes. Same guest machine boot for about 4-7 seconds on virtualbox/vmware
This is very unlikely to do with the kernel. Qemu (which you're using to run your guest -- kvm is just in-kernel module) will emulate whole machine including CPU in case it can't access /dev/kvm or if you requested to not use it in the first place. If the problem is due to lack of access to /dev/kvm, qemu tells you about that on stderr. So far, you haven't provided any of: - qemu version - qemu command line - messages generated by qemu And for this issue, please also provide output of info kvm from qemu monitor. If kvm is enabled, it should tell you just that -- "enabled". If not, it'll tell you "disabled" and this means that your guest is running in full emulation mode. Without that, this bugreport is not very useful. Especially having in mind that this setup works for lots of other people just fine.
cmdline: sudo kvm -enable-kvm -cpu 'host' -smp '2,cores=2,threads=2,sockets=1' -m 512M -vga vmware -usbdevice tablet -soundhw es1370 -drive file=test-amd64.img,if=virtio,cache=none -netdev type=tap,script=/etc/qemu-ifup,id=net0 -device virtio-net-pci,netdev=net0 -balloon virtio -daemonize -name 'Gentoo Hardened 64 20120207' version: qemu -version QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard info: (qemu) info kvm kvm support: enabled (qemu) Host dmesg kvm only : roko__@ArchXFCE64(11:34:21-12-02-10)( ~ pts/1 )# dmesg | grep kvm [12347.567390] kvm: Nested Virtualization enabled [12374.734909] kvm: unreliable cycle conversion on adjustable rate TSC [12451.916934] kvm: 13119: cpu0 unhandled rdmsr: 0xc001100d [12451.916959] kvm: 13119: cpu0 unhandled rdmsr: 0xc0010112 [12452.286858] kvm: 13119: cpu0 unhandled rdmsr: 0xc0010001 [16345.719346] kvm: 23604: cpu0 unhandled rdmsr: 0xc0010112 [16346.104688] kvm: 23604: cpu0 unhandled rdmsr: 0xc0010001 [23694.763207] kvm: 4941: cpu0 unhandled rdmsr: 0xc0010112 [23694.969082] kvm: 4941: cpu0 unhandled rdmsr: 0xc0010001 [23830.158470] kvm: 4941: cpu0 unhandled rdmsr: 0xc0010112 [23830.360768] kvm: 4941: cpu0 unhandled rdmsr: 0xc0010001 [23882.579070] kvm: 5419: cpu0 unhandled rdmsr: 0xc0010112 [23882.779127] kvm: 5419: cpu0 unhandled rdmsr: 0xc0010001 [24081.106328] kvm: 5770: cpu0 unhandled rdmsr: 0xc0010112 [24081.439772] kvm: 5770: cpu0 unhandled rdmsr: 0xc0010001 [24478.792921] kvm: 7706: cpu0 unhandled rdmsr: 0xc0010112 [24478.986782] kvm: 7706: cpu0 unhandled rdmsr: 0xc0010001 [24931.920782] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable [24942.147749] kvm: 8966: cpu0 unhandled rdmsr: 0xc0010112 [24942.301014] kvm: 8966: cpu0 unhandled rdmsr: 0xc0010001 [27046.175073] kvm: 15606: cpu0 unhandled rdmsr: 0xc0010112 [27046.332410] kvm: 15606: cpu0 unhandled rdmsr: 0xc0010001 [28459.859245] kvm: 23994: cpu0 unhandled rdmsr: 0xc0010112 [28460.058113] kvm: 23994: cpu0 unhandled rdmsr: 0xc0010001 [30892.864676] kvm: 6341: cpu0 unhandled rdmsr: 0xc0010001 [31438.995057] kvm: 11147: cpu0 unhandled rdmsr: 0xc0010112 [31439.214065] kvm: 11147: cpu0 unhandled rdmsr: 0xc0010001 [32561.185944] kvm: 20349: cpu0 unhandled rdmsr: 0xc0010112 [32561.448391] kvm: 20349: cpu0 unhandled rdmsr: 0xc0010001 [33633.457536] kvm: 28893: cpu0 unhandled rdmsr: 0xc0010112 [33633.801818] kvm: 28893: cpu0 unhandled rdmsr: 0xc0010001 [33879.382766] kvm: 30299: cpu0 unhandled rdmsr: 0xc0010112 [33879.577796] kvm: 30299: cpu0 unhandled rdmsr: 0xc0010001 [34654.862032] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010112 [34655.234667] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010001 [36724.031803] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010112 [36724.147844] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010001 [36836.657695] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010112 [36836.896359] kvm: 32668: cpu0 unhandled rdmsr: 0xc0010001 [36891.644691] kvm: 8665: cpu0 unhandled rdmsr: 0xc0010112 [36891.778808] kvm: 8665: cpu0 unhandled rdmsr: 0xc0010001 [37065.796742] kvm: 9545: cpu0 unhandled rdmsr: 0xc0010112 [37065.951394] kvm: 9545: cpu0 unhandled rdmsr: 0xc0010001 [37198.255255] kvm: 10184: cpu0 unhandled rdmsr: 0xc0010112 [37198.385243] kvm: 10184: cpu0 unhandled rdmsr: 0xc0010001 [37531.276399] kvm: 11308: cpu0 unhandled rdmsr: 0xc0010112 [37531.404946] kvm: 11308: cpu0 unhandled rdmsr: 0xc0010001 [37633.200338] kvm: 12026: cpu0 unhandled rdmsr: 0xc0010112 [37633.335279] kvm: 12026: cpu0 unhandled rdmsr: 0xc0010001 [37974.339382] kvm: 13679: cpu0 unhandled rdmsr: 0xc0010112 [37974.463314] kvm: 13679: cpu0 unhandled rdmsr: 0xc0010001 [37998.836470] kvm: 14017: cpu0 unhandled rdmsr: 0xc0010112 [37998.952253] kvm: 14017: cpu0 unhandled rdmsr: 0xc0010001 [38194.260773] kvm: 14756: cpu0 unhandled rdmsr: 0xc0010112 [38194.400660] kvm: 14756: cpu0 unhandled rdmsr: 0xc0010001 [40609.026332] kvm: 23045: cpu0 unhandled rdmsr: 0xc0010001 [41580.924327] kvm: 28276: cpu0 unhandled rdmsr: 0xc0010112 [41581.040324] kvm: 28276: cpu0 unhandled rdmsr: 0xc0010001 [89588.419228] kvm: 5045: cpu0 unhandled rdmsr: 0xc0010112 [89588.558034] kvm: 5045: cpu0 unhandled rdmsr: 0xc0010001 [126250.670121] kvm: 27098: cpu0 unhandled rdmsr: 0xc0010112 [126250.800737] kvm: 27098: cpu0 unhandled rdmsr: 0xc0010001 [128187.690739] kvm: 27098: cpu0 unhandled rdmsr: 0xc0010112 [128187.821740] kvm: 27098: cpu0 unhandled rdmsr: 0xc0010001 [128452.392446] kvm: 1746: cpu0 unhandled rdmsr: 0xc0010112 [128452.524364] kvm: 1746: cpu0 unhandled rdmsr: 0xc0010001 [128642.028625] kvm: 1746: cpu0 unhandled rdmsr: 0xc0010112 [128642.165803] kvm: 1746: cpu0 unhandled rdmsr: 0xc0010001 [129900.176718] kvm: 6916: cpu0 unhandled rdmsr: 0xc0010112 [129900.301721] kvm: 6916: cpu0 unhandled rdmsr: 0xc0010001 [130077.225227] kvm: 8083: cpu0 unhandled rdmsr: 0xc0010112 [130077.355747] kvm: 8083: cpu0 unhandled rdmsr: 0xc0010001 [130251.383970] kvm: 8899: cpu0 unhandled rdmsr: 0xc001100d [130251.383995] kvm: 8899: cpu0 unhandled rdmsr: 0xc0010112 [130251.526014] kvm: 8899: cpu0 unhandled rdmsr: 0xc0010001 [130251.552576] kvm: 8899: cpu1 unhandled rdmsr: 0xc001100d [172942.439354] kvm: 6960: cpu0 unhandled rdmsr: 0xc001100d [172942.439379] kvm: 6960: cpu0 unhandled rdmsr: 0xc0010112 [172942.559674] kvm: 6960: cpu0 unhandled rdmsr: 0xc0010001 [172942.584961] kvm: 6960: cpu1 unhandled rdmsr: 0xc001100d [173128.821234] kvm: 7982: cpu0 unhandled rdmsr: 0xc0010112 [173128.952489] kvm: 7982: cpu0 unhandled rdmsr: 0xc0010001 [202490.969575] kvm: 26305: cpu0 unhandled rdmsr: 0xc0010112 [202491.091029] kvm: 26305: cpu0 unhandled rdmsr: 0xc0010001 [203118.612164] kvm: 28617: cpu0 unhandled rdmsr: 0xc0010112 [203118.753413] kvm: 28617: cpu0 unhandled rdmsr: 0xc0010001 [203530.933754] kvm: 30072: cpu0 unhandled rdmsr: 0xc0010112 [203531.058048] kvm: 30072: cpu0 unhandled rdmsr: 0xc0010001 [203966.434833] kvm: 32198: cpu0 unhandled rdmsr: 0xc0010112 [203966.561245] kvm: 32198: cpu0 unhandled rdmsr: 0xc0010001 [204582.016825] kvm: 32198: cpu0 unhandled rdmsr: 0xc0010112 [204582.152043] kvm: 32198: cpu0 unhandled rdmsr: 0xc0010001 [204863.395916] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010112 [204863.530150] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010001 [207390.936205] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010112 [207391.077543] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010001 [255644.169559] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010112 [255644.310790] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010001 [256444.141527] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010112 [256444.270798] kvm: 3269: cpu0 unhandled rdmsr: 0xc0010001 [256684.553828] kvm: 30582: cpu0 unhandled rdmsr: 0xc0010112 [256684.678749] kvm: 30582: cpu0 unhandled rdmsr: 0xc0010001 [259699.726226] kvm: 30582: cpu0 unhandled rdmsr: 0xc0010112 [259699.934852] kvm: 30582: cpu0 unhandled rdmsr: 0xc0010001 roko__@ArchXFCE64(11:34:28-12-02-10)( ~ pts/1 )#
Still no response there ..
Try without -vga vmware. If still slow provide kvm trace please (http://www.linux-kvm.org/page/Tracing)
@Gleb i tryed many variants cirrus std vmware vga-s no diferences.
@Gleb trace-cmd cannot install ********************************* ** No docbook.xsl is installed ** ** Can't make man pages ** ********************************* extra/docbook-xsl 1.76.1-2 [installed] XML stylesheets for Docbook-xml transformations
(In reply to comment #7) > @Gleb > > trace-cmd cannot install > > ********************************* > ** No docbook.xsl is installed ** > ** Can't make man pages ** > ********************************* > > extra/docbook-xsl 1.76.1-2 [installed] > XML stylesheets for Docbook-xml transformations Please resolve the problem and try again. Which host distribution are you using?
Hm .. trace-cmd record -b 20000 -e kvm trace-cmd: No such file or directory opening '/sys/kernel/debug/tracing/tracing_on' this directory doesn't exist.
(In reply to comment #8) > (In reply to comment #7) > > @Gleb > > > > trace-cmd cannot install > > > > ********************************* > > ** No docbook.xsl is installed ** > > ** Can't make man pages ** > > ********************************* > > > > extra/docbook-xsl 1.76.1-2 [installed] > > XML stylesheets for Docbook-xml transformations > > Please resolve the problem and try again. Which host distribution are you > using? Host distribution is archlinux. I write answer to trace-cmd, missing directory.
(In reply to comment #9) > Hm .. > > trace-cmd record -b 20000 -e kvm > trace-cmd: No such file or directory > opening '/sys/kernel/debug/tracing/tracing_on' > > this directory doesn't exist. mount -t debugfs none /sys/kernel/debug/
(In reply to comment #11) > (In reply to comment #9) > > Hm .. > > > > trace-cmd record -b 20000 -e kvm > > trace-cmd: No such file or directory > > opening '/sys/kernel/debug/tracing/tracing_on' > > > > this directory doesn't exist. > > mount -t debugfs none /sys/kernel/debug/ mount -t debugfs none /sys/kernel/debug/ mount: none already mounted or /sys/kernel/debug/ busy mount: according to mtab, nodev is already mounted on /sys/kernel/debug Allready mounted. Bu this directory doesn't exist, i have those dirs: /sys/kernel/debug acpi bdi dri hid ieee80211 kvm mce suspend_stats usb virtio-ports wakeup_sources x86
(In reply to comment #12) > Allready mounted. Bu this directory doesn't exist, i have those dirs: > > /sys/kernel/debug > acpi bdi dri hid ieee80211 kvm mce suspend_stats usb virtio-ports > wakeup_sources x86 Is this ancient kernel? Or have you disabled tracing during kernel compile? If later you will have to rebuild the kernel with tracing enabled. This is the only way to see what is happening.
(In reply to comment #13) > (In reply to comment #12) > > Allready mounted. Bu this directory doesn't exist, i have those dirs: > > > > /sys/kernel/debug > > acpi bdi dri hid ieee80211 kvm mce suspend_stats usb virtio-ports > > wakeup_sources x86 > > Is this ancient kernel? Or have you disabled tracing during kernel compile? > If > later you will have to rebuild the kernel with tracing enabled. This is the > only way to see what is happening. Its my kernel not default ARCH kernel. zgrep TRACE /proc/config.gz CONFIG_STACKTRACE_SUPPORT=y # CONFIG_RCU_TRACE is not set # CONFIG_TREE_RCU_TRACE is not set CONFIG_HAVE_ARCH_TRACEHOOK=y # CONFIG_TRACE_SINK is not set CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_USER_STACKTRACE_SUPPORT=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_HAVE_SYSCALL_TRACEPOINTS=y # CONFIG_FTRACE is not set CONFIG_HAVE_MMIOTRACE_SUPPORT=y i'm not sure which option need to be enabled?!
(In reply to comment #14) > (In reply to comment #13) > > (In reply to comment #12) > > > Allready mounted. Bu this directory doesn't exist, i have those dirs: > > > > > > /sys/kernel/debug > > > acpi bdi dri hid ieee80211 kvm mce suspend_stats usb > virtio-ports > > > wakeup_sources x86 > > > > Is this ancient kernel? Or have you disabled tracing during kernel compile? > If > > later you will have to rebuild the kernel with tracing enabled. This is the > > only way to see what is happening. > > Its my kernel not default ARCH kernel. > > zgrep TRACE /proc/config.gz > CONFIG_STACKTRACE_SUPPORT=y > # CONFIG_RCU_TRACE is not set > # CONFIG_TREE_RCU_TRACE is not set > CONFIG_HAVE_ARCH_TRACEHOOK=y > # CONFIG_TRACE_SINK is not set > CONFIG_TRACE_IRQFLAGS_SUPPORT=y > CONFIG_USER_STACKTRACE_SUPPORT=y > CONFIG_HAVE_FUNCTION_TRACER=y > CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y > CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y > CONFIG_HAVE_DYNAMIC_FTRACE=y > CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y > CONFIG_HAVE_SYSCALL_TRACEPOINTS=y > # CONFIG_FTRACE is not set > CONFIG_HAVE_MMIOTRACE_SUPPORT=y > zgrep TRACING /proc/config.gz CONFIG_TRACING_SUPPORT=y > i'm not sure which option need to be enabled?!
(In reply to comment #15) > > zgrep TRACING /proc/config.gz > CONFIG_TRACING_SUPPORT=y > You need CONFIG_TRACING/CONFIG_FTRACE and others.
hello again! roko__@ArchXFCE64(16:47:28-12-02-12)( /mnt/home/roko__-backup/kvm pts/6 )# trace-cmd report version = 6 cound not load plugin '/usr/local/share/trace-cmd/plugins/plugin_python.so' /usr/local/share/trace-cmd/plugins/plugin_python.so: undefined symbol: PyString_FromString File possibly truncated. Need at least 1654784, but file size is 1654292. trace-cmd: Invalid argument failed to init data And the trace.dat file is in attachments.
Created attachment 72360 [details] sudo trace-cmd record -b 20000 -e kvm trace.dat from roko__@ArchXFCE64(16:46:25-12-02-12)( /mnt/home/roko__-backup/kvm pts/6 )# sudo trace-cmd record -b 20000 -e kvm /sys/kernel/debug/tracing/events/kvm/filter /sys/kernel/debug/tracing/events/*/kvm/filter Hit Ctrl^C to stop recording ^CKernel buffer statistics: Note: "entries" are the entries left in the kernel ring buffer and are not recorded in the trace data. They should all be zero. CPU: 0 entries: 0 overrun: 0 commit overrun: 0 bytes: 0 oldest event ts: 18446612134.805802 now ts: 3383.418379 CPU: 1 entries: 0 overrun: 0 commit overrun: 0 bytes: 0 oldest event ts: 142390.782329 now ts: 3383.418586 CPU0 data recorded at offset=0x194000 0 bytes in size CPU1 data recorded at offset=0x194000 0 bytes in size roko__@ArchXFCE64(16:47:23-12-02-12)( /mnt/home/roko__-backup/kvm pts/6 )#
(In reply to comment #17) > hello again! > > roko__@ArchXFCE64(16:47:28-12-02-12)( /mnt/home/roko__-backup/kvm pts/6 )# > trace-cmd report > version = 6 > cound not load plugin '/usr/local/share/trace-cmd/plugins/plugin_python.so' > /usr/local/share/trace-cmd/plugins/plugin_python.so: undefined symbol: > PyString_FromString > > File possibly truncated. Need at least 1654784, but file size is 1654292. > trace-cmd: Invalid argument > failed to init data > > And the trace.dat file is in attachments. It is corrupted for me too. Have you compiled trace-cmd by yourself? If yes have you ran "make install" before using it?
i compilled it from AUR, maybe package is broken
Can you please compile trace-cmd from its git [1] (do "make all_cmd install_cmd"; install part is important) and try getting trace with it? If this will not work I will guide you how to take a trace using debugfs directly. [1] git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
Created attachment 72363 [details] trace-cmd report
What guest did during this trace? Can you provide "info pci" monitor output pls?
Please run 'perf top' in the host and report the output (while tracing is disabled).
(In reply to comment #23) > What guest did during this trace? Can you provide "info pci" monitor output > pls? can't see full output from this command
Created attachment 72365 [details] info pci
and there soon will be video capture with 'perf top' http://vbox7.com/play:199e9ede30
(In reply to comment #27) > and there soon will be video capture with 'perf top' > > http://vbox7.com/play:199e9ede30 Run it while the guest is also running.
(In reply to comment #28) > (In reply to comment #27) > > and there soon will be video capture with 'perf top' > > > > http://vbox7.com/play:199e9ede30 > > Run it while the guest is also running. Good Morning! There will be video http://vbox7.com/play:7128f03f1f after some momments.
Disable ksm, and build with debug information so we get useful information instead of hex addresses.
any responses ? with/without ksm no change.
Can you run perf without ksm? In your previous perf invocation ksm was at the top, so lets see what is it there now.
Supposing it was working good with an old version of kvm: Why don't you just do a git bisect? qemu-kvm is very quick to compile and install. Start kvm with -snapshot so you can kill it anytime; and just by visual inspection at boot you should be able to determine if it is fast or slow and proceed with next step of bisect. In a short time you would probably solve this.
(In reply to comment #33) > Supposing it was working good with an old version of kvm: > Why don't you just do a git bisect? qemu-kvm is very quick to compile and > install. Start kvm with -snapshot so you can kill it anytime; and just by > visual inspection at boot you should be able to determine if it is fast or > slow > and proceed with next step of bisect. In a short time you would probably > solve > this. not that fast compilling, it eat my memory durring compille process and going to megaswapping.. 100% CPU RAM SWAP usage when compilling kvm.
No changes with git version of qemu-kvm Only see when work guest cpu usage is out of range, about 120% - 150%