Bug 45931 - Nested Virt: VMX can't be initialized in L1 Xen ("Xen on KVM")
Summary: Nested Virt: VMX can't be initialized in L1 Xen ("Xen on KVM")
Alias: None
Product: Virtualization
Classification: Unclassified
Component: kvm (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: virtualization_kvm
Depends on:
Blocks: 94971 53601
  Show dependency tree
Reported: 2012-08-14 06:40 UTC by Jay Ren
Modified: 2015-03-17 03:53 UTC (History)
7 users (show)

See Also:
Kernel Version: 3.6-RC1
Tree: Mainline
Regression: No

xl-dmesg-in-L1-Xen.log (7.38 KB, text/plain)
2012-08-14 06:40 UTC, Jay Ren

Description Jay Ren 2012-08-14 06:40:10 UTC
Created attachment 77661 [details]

Hi folks,
I did some basic testing on nested virtualization on Intel x86-64 platform.
Will KVM support Xen as L1 guest in nested virtualization ?

When I tried "Xen on KVM" mode, I found VMX can't be initialized in L1 Xen hypervisor.
I tried both "-cpu host" and "-cpu qemu64,+vmx" parameters in qemu-kvm command line.
-------some log in 'xl dmesg' command line in L1 Xen ---
(XEN) VMX: CPU0 has insufficent CPU-Based Exec Control (b299868c but requires min 2299968c)
(XEN) VMX: CPU0 has insufficent VMExit Control (00000200 but requires min 00008200)
(XEN) VMX: failed to initialise.
------more details in the attached "xl-dmesg-L1.log"----
In Xen source code, I found the following related to the mentioned error.
#define CPU_BASED_RDTSC_EXITING               0x00001000
#define VM_EXIT_ACK_INTR_ON_EXIT        0x00008000

Avi confirmed there're two issues, and fixed 1st one (RDTSC) by the following patch.

All processors that support VMX have that feature, and guests (Xen) depend on
it.  As we already implement it, advertize it to the guest.

Signed-off-by: Avi Kivity <avi@redhat.com>
 arch/x86/kvm/vmx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index cc8ad98..8092f25 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -1990,7 +1990,7 @@ static __init void nested_vmx_setup_ctls_msrs(void)
 	 * We can allow some features even when not supported by the
Comment 1 Nadav Har'El 2013-02-11 15:10:48 UTC
After Avi fixed the missing support CPU_BASED_RDTSC_EXITING, what remains (at least) is missing support for VM_EXIT_ACK_INTR_ON_EXIT.

See the following thread on how this support might be added:
Comment 2 Zhou, Chao 2014-05-20 07:33:08 UTC
kvm.git + qemu.git:198c74f4_a41b2c99
host kernel :3.15.0-rc1
test on Romley_EP, crete L1(xen on kvm) guest with "-cpu host", then create L2 guest, L1 guest can boot up.

some log with command "xl dmesg" in L1 guest
(XEN) Platform timer is 100.000MHz HPET
(XEN) Allocated console ring of 32 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
Comment 3 Paolo Bonzini 2014-05-20 11:02:40 UTC
Patches were committed to kvm/next and will be included in 3.16.

Note You need to log in before you can comment on or make changes to this bug.