Bug 42779

Summary: KVM domain hangs after loading initrd with Xenomai kernel
Product: Virtualization Reporter: madengineer10
Component: kvmAssignee: virtualization_kvm
Status: RESOLVED CODE_FIX    
Severity: normal CC: alan, avi, florian, gleb, jan.kiszka
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 3.0.0-15 Subsystem:
Regression: No Bisected commit-id:
Attachments: Configuration of the guest kernel
Result of 'registers info' and 'x/30i $eip' after fault
Last 10k lines of a trace showing the fault
Register state and code disassembly at failure point with qemu 1.0
Register state and code disassembly at failure point, kernel 3.3rc5 with patches
Last 10k lines of a trace showing the fault, kernel 3.3rc5 with patches

Description madengineer10 2012-02-16 05:45:29 UTC
Attempting to boot a 32 bit Debian guest with a Xenomai kernel inside KVM causes it to hang and spin (using 1 full CPU core) after loading the initrd, as determined by serial console output. The only error message is "KVM internal error. Suberror: 1"/"emulation failure". Booting a regular Debian kernel succeeds, as does running the Xenomai kernel with software emulation (-no-kvm).

Info:
CPU: Intel Core i7-2670QM
Emulator: qemu-kvm 0.14.1
Host kernel: 3.0.0-15 (Ubuntu build), x86_64
Guest OS: Debian Squeeze, kernel.org 2.6.37 kernel with Xenomai 2.6.0 (config attached)
Qemu command: kvm -M pc-0.14 -enable-kvm -m 1024 -drive file=/var/lib/libvirt/images/eve.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=21,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:b5:f4:00,bus=pci.0,addr=0x3 -chardev stdio,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vga cirrus
Effects of flags: Adding one or both of --no-kvm-irqchip or --no-kvm-pit has no apparent effect. Adding --no-kvm appears to correct the problem, at the cost of performance due to using the software emulator.
Comment 1 madengineer10 2012-02-16 05:46:07 UTC
Created attachment 72393 [details]
Configuration of the guest kernel
Comment 2 madengineer10 2012-02-16 05:47:18 UTC
Created attachment 72394 [details]
Result of 'registers info' and 'x/30i $eip' after fault
Comment 3 madengineer10 2012-02-16 05:57:25 UTC
Couldn't attach the trace I recorded of the fault occurring since it's 3 MB compressed with xz, bigger still with other formats. I can email it if it will be useful.
Comment 4 madengineer10 2012-02-16 06:32:48 UTC
Same problem occurs with qemu-kvm 1.0 from https://launchpad.net/~bderzhavets/+archive/lib-usbredir39:

$ sudo kvm -M pc-1.0 -enable-kvm -m 1024 -drive file=/var/lib/libvirt/images/eve.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=21,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:b5:f4:00,bus=pci.0,addr=0x3 -chardev vc,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vga cirrus
kvm: -netdev tap,fd=21,id=hostnet0: TUNGETIFF ioctl() failed: Bad file descriptor
TUNSETOFFLOAD ioctl() failed: Bad file descriptor
kvm: -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:b5:f4:00,bus=pci.0,addr=0x3: pci_add_option_rom: failed to find romfile "pxe-e1000.rom"
KVM internal error. Suberror: 1
emulation failure
EAX=f6810000 EBX=0000003e ECX=0000003e EDX=c00b8000
ESI=c00b8000 EDI=c15b0000 EBP=c15b1f74 ESP=c15b1f58
EIP=c1228905 EFL=00010206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0000 00000000 ffffffff 00000000
GS =0000 00000000 ffffffff 00000000
LDT=0000 00000000 ffffffff 00000000
TR =0080 c15b6300 0000206b 00008b00 DPL=0 TSS32-busy
GDT=     c15b3000 000000ff
IDT=     c15b2000 000007ff
CR0=80050033 CR2=ffee4000 CR3=01663000 CR4=00000690
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=8e 2b 01 00 00 8b 4d f0 89 f2 8b 45 ec 0f 0d 82 40 01 00 00 <0f> 6f 02 0f 6f 4a 08 0f 6f 52 10 0f 6f 5a 18 0f 7f 00 0f 7f 48 08 0f 7f 50 10 0f 7f 58 18
Comment 5 Gleb 2012-02-16 07:15:49 UTC
(In reply to comment #3)
> Couldn't attach the trace I recorded of the fault occurring since it's 3 MB
> compressed with xz, bigger still with other formats. I can email it if it
> will
> be useful.

Can you do "tail -10000" on it and attach it here?
Comment 6 madengineer10 2012-02-16 07:22:41 UTC
Created attachment 72395 [details]
Last 10k lines of a trace showing the fault

Per Gleb's request
Comment 7 Gleb 2012-02-16 07:43:11 UTC
Have you installed trace-cmd before capturing the trace? It failed to parse kvm events. qemu haven't paused the guest after emulation error (looks like a bug), so 'x/30i $eip' output is not useful either. Can you do 'x/30i 0xXXX' where XXX is the address in EIP from register dump you see after instruction emulation failure message (c1228905 in your output from comment #4)?
Comment 8 madengineer10 2012-02-16 07:56:03 UTC
Not sure what you mean by installing trace-cmd before capturing the trace--I did do that, otherwise I wouldn't have had a trace-cmd to run. The package version is trace-cmd 1.0.3-0ubuntu1 if that helps. I tried running it again against qemu 1.0 (the last one was for qemu 0.14), still contained a bunch of [FAILED TO PARSE] messages.

Attaching the output you requested separately.
Comment 9 madengineer10 2012-02-16 07:57:35 UTC
Created attachment 72397 [details]
Register state and code disassembly at failure point with qemu 1.0
Comment 10 Gleb 2012-02-16 08:04:22 UTC
(In reply to comment #8)
> Not sure what you mean by installing trace-cmd before capturing the trace--I
> did do that, otherwise I wouldn't have had a trace-cmd to run. The package
I meant that if you compile it by yourself you need to run "make install" instead of running it from where it was compiled. Otherwise it does not find plugins. If you install it from package manager this is not relevant of course.

> version is trace-cmd 1.0.3-0ubuntu1 if that helps. I tried running it again
> against qemu 1.0 (the last one was for qemu 0.14), still contained a bunch of
> [FAILED TO PARSE] messages.
> 
Something wrong with ubuntu trace-cmd then. Either it does not have kvm plugin or it is too old.
Comment 11 Jan Kiszka 2012-02-16 08:31:18 UTC
(In reply to comment #4)
> Code=8e 2b 01 00 00 8b 4d f0 89 f2 8b 45 ec 0f 0d 82 40 01 00 00 <0f> 6f 02
> 0f
> 6f 4a 08 0f 6f 52 10 0f 6f 5a 18 0f 7f 00 0f 7f 48 08 0f 7f 50 10 0f 7f 58 18

This is "movq (%rdx),%mm0". Possibly still unsupported by the emulator.
Comment 12 Gleb 2012-02-16 08:40:37 UTC
Can you try disabling CONFIG_FB in your guest kernel?
Comment 13 Avi Kivity 2012-02-16 11:40:22 UTC
Indeed the emulator doesn't support mmx yet.  But why is the kernel using mmx instead of sse?
Comment 14 Jan Kiszka 2012-02-16 11:51:05 UTC
Because the selected CPU type doesn't support SSE? Didn't check the config yet, but I bet that's the reason.
Comment 15 madengineer10 2012-02-16 16:48:56 UTC
I set the kernel up for an AMD Geode LX800 (CONFIG_MGEODE_LX), which does supposedly support SSE. No idea why it would have built using MMX if that is now deprecated in favor of SSE.

I can try disabling the framebuffer (CONFIG_FB/CONFIG_FB_GEODE/CONFIG_FB_GEODE_LX) for testing purposes; unfortunately this wouldn't work for the real thing, since we need it for the Qt-based user interface.
Comment 16 madengineer10 2012-02-16 16:51:10 UTC
Background: I'm using KVM as a test platform for code destined for a panel PC using a Geode LX800.
Comment 17 Jan Kiszka 2012-02-16 17:24:34 UTC
(In reply to comment #15)
> I set the kernel up for an AMD Geode LX800 (CONFIG_MGEODE_LX), which does
> supposedly support SSE. No idea why it would have built using MMX if that is
> now deprecated in favor of SSE.

At least not according to the Linux configuration: MGEODE_LX only sets X86_USE_3DNOW, thus MMX.
Comment 18 Avi Kivity 2012-02-16 17:30:47 UTC
As a workaround you can use -cpu blah,-3dnow.  But we'll have to implement mmx movq.
Comment 19 madengineer10 2012-02-16 22:01:38 UTC
Doesn't seem to help. I used -cpu core2duo,-3dnow (since core2duo is the name that libvirt, my usual launcher for qemu, has been using), failed on the same movq. Also tried -cpu core2duo,-3dnow,-mmx, same results. I'll see if I can build a kernel with the Geode framebuffer driver but without setting the CPU to Geode (thus eliminating the use of MMX).
Comment 20 Avi Kivity 2012-03-21 18:21:33 UTC
Please try

  git://git.kernel.org/pub/scm/virt/kvm/kvm.git emulator-movq

this emulates the two movq instructions used.  There may be other unemulated instructions though.
Comment 21 madengineer10 2012-03-31 20:10:38 UTC
Sorry for the delay, I've been really busy lately. Is this kernel reasonably safe to run on my main machine? I'm not worried about it crashing, but if it takes a filesystem with it...
Comment 22 Avi Kivity 2012-04-01 14:15:51 UTC
No guarantees of course, but this is just 3.3-rc5 with a few patches on top.  It should be fine for a short test.
Comment 23 madengineer10 2012-04-01 23:38:21 UTC
Now it dies in a different place, still with a movq instruction (movq   (%edx),%mm0). I think the difference is because of a configuration change or package update since the last test, because the fault happens at the same address and same instruction under both the patched 3.3rc5 and 3.0.20. Unfortunately I've no idea what the configuration change might have been, and I didn't think to save a VM disk image from the previous tests.

I also got a more recent trace-cmd built and installed, so there is usable trace output now. Still huge, so I'll post the last 10k lines as before.
Comment 24 madengineer10 2012-04-01 23:39:47 UTC
Created attachment 72788 [details]
Register state and code disassembly at failure point, kernel 3.3rc5 with patches
Comment 25 madengineer10 2012-04-01 23:40:21 UTC
Created attachment 72789 [details]
Last 10k lines of a trace showing the fault, kernel 3.3rc5 with patches
Comment 26 Avi Kivity 2012-04-02 09:12:38 UTC
Strange, looks like the patches did not take effect at all.  The opcode is 0f 6f 02, which should have been decoded as movq.
Comment 27 Florian Mickler 2012-07-01 09:42:18 UTC
A patch referencing this bug report has been merged in Linux v3.5-rc1:

commit e59717550e5cf0e7159c5b7af1d1ead35fef49dd
Author: Avi Kivity <avi@redhat.com>
Date:   Mon Apr 9 18:40:03 2012 +0300

    KVM: x86 emulator: implement MMX MOVQ (opcodes 0f 6f, 0f 7f)