Bug 73331

Summary:	Nested Virtualization, L2 cannot boot up on Ivybridge and Haswell
Product:	Virtualization	Reporter:	Robert Ho (Robert.Hu)
Component:	kvm	Assignee:	virtualization_kvm
Status:	VERIFIED CODE_FIX
Severity:	normal	CC:	bonzini, bsd, chao.zhou, freyes, jan.kiszka, sheshas
Priority:	P1
Hardware:	x86-64
OS:	Linux
Kernel Version:	3.14.0-rc3	Subsystem:
Regression:	No	Bisected commit-id:

Description Robert Ho 2014-04-01 06:47:08 UTC

Environment:
------------
Host OS (ia32/ia32e/IA64):ia32e
Guest OS (ia32/ia32e/IA64):ia32e
Guest OS Type (Linux/Windows):Linux
kvm.git Commit:7227fc0666606b0df2c0d2966a7f4859b01bdf74
qemu.git Commit: 8648fcd52a9bcc2aa415cbe87b7c636e545acb38
Host Kernel Version:3.14.0-rc3
Hardware:Ivytown_EP, Grantley-EP

Bug detailed description:
--------------------------
create L2 guest, L2 guest cannot boot up on Ivytown.


Reproduce steps:
----------------
1.create L1 guests:
qemu-system-x86_64 --enable-kvm -m 10240 -smp 8 -net
nic,macaddr=00:12:45:67:2B:1C -net tap,script=/etc/kvm/qemu-ifup
nested-kvm-rhel6u4.qcow -cpu host
2. create L2 guests:
qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -net none rhel6u4.qcow


Current result:
----------------
L2 guest cannot boot up fine

Expected result:
----------------
L2 guest boot up fine.

Basic root-causing log:
----------------------
[root@rhel6u4 ~]# qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -net none rhel6u4.qcow
VNC server running on `::1:5900'
KVM: entry failed, hardware error 0x0
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000623
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 0000f300
SS =0000 00000000 0000ffff 0000f300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff 66 89 c8 66 5b 66 5e 66 5f 66 5d 66 c3 fb f4 fa fc 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Comment 1 Bandan Das 2014-04-04 15:57:34 UTC

(In reply to Robert.Hu from comment #0)
> 
> Basic root-causing log:
> ----------------------
> [root@rhel6u4 ~]# qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -net none
> rhel6u4.qcow
> VNC server running on `::1:5900'
> KVM: entry failed, hardware error 0x0
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000623
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=0000fff0 EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 0000f300
> SS =0000 00000000 0000ffff 0000f300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT=     00000000 0000ffff
> IDT=     00000000 0000ffff
> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=ff 66 89 c8 66 5b 66 5e 66 5f 66 5d 66 c3 fb f4 fa fc 66 c3 <ea> 5b e0
> 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00 00

Can you post your dmesg too ?

Comment 2 Shesha 2014-04-17 18:57:32 UTC

Same issue when using on sandy bridge on L0 host. This happens with Sandybridge and kernel version 3.13.0-23-generic #45-Ubuntu (Trusty). However, this works on the exact same hardware (same physical L0 host) with 3.2.0-23-generic #36-Ubuntu (Precise).

ON L0
======
cat /proc/cpuinfo | head -20
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
stepping	: 4
microcode	: 0x416
cpu MHz		: 3000.063
cache size	: 25600 KB
physical id	: 0
siblings	: 20
core id		: 0
cpu cores	: 10
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms


DMESG: [504490.129124] nested_vmx_exit_handled failed vm entry 7

Contents of /sys/module/kvm_intel/parameters:
emulate_invalid_guest_state :Y
enable_apicv :Y
enable_shadow_vmcs :N
ept :Y
eptad :N
fasteoi :Y
flexpriority :Y
nested :Y
ple_gap :128
ple_window :4096
unrestricted_guest :Y
vmm_exclusive :Y
vpid :Y

On L1
======

KVM: entry failed, hardware error 0x0
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000623
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=21 83 66 ba 01 49 03 00 66 89 d8 66 5b 66 5e e9 d7 7e 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Comment 3 Zhou, Chao 2014-07-16 07:45:52 UTC

kvm.git + qemu.git:9f6226a7_0e162974
kernel version: 3.16.0-rc1
test on Ivytown_EP
after create L2 guest, L2 guest can boot up.