Bug 45931

Summary: Nested Virt: VMX can't be initialized in L1 Xen ("Xen on KVM")
Product: Virtualization Reporter: Jay Ren (yongjie.ren)
Component: kvmAssignee: virtualization_kvm
Status: RESOLVED CODE_FIX    
Severity: normal CC: avi, bonzini, chao.zhou, dustymabe, gianluca.amato.74, nyh, omer
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 3.6-RC1 Subsystem:
Regression: No Bisected commit-id:
Bug Depends on:    
Bug Blocks: 94971, 53601    
Attachments: xl-dmesg-in-L1-Xen.log

Description Jay Ren 2012-08-14 06:40:10 UTC
Created attachment 77661 [details]
xl-dmesg-in-L1-Xen.log

Hi folks,
I did some basic testing on nested virtualization on Intel x86-64 platform.
Will KVM support Xen as L1 guest in nested virtualization ?

When I tried "Xen on KVM" mode, I found VMX can't be initialized in L1 Xen hypervisor.
I tried both "-cpu host" and "-cpu qemu64,+vmx" parameters in qemu-kvm command line.
-------some log in 'xl dmesg' command line in L1 Xen ---
(XEN) VMX: CPU0 has insufficent CPU-Based Exec Control (b299868c but requires min 2299968c)
(XEN) VMX: CPU0 has insufficent VMExit Control (00000200 but requires min 00008200)
(XEN) VMX: failed to initialise.
------more details in the attached "xl-dmesg-L1.log"----
In Xen source code, I found the following related to the mentioned error.
"xen-unstable.hg/xen/include/asm-x86/hvm/vmx/vmcs.h"
#define CPU_BASED_RDTSC_EXITING               0x00001000
#define VM_EXIT_ACK_INTR_ON_EXIT        0x00008000


----
Avi confirmed there're two issues, and fixed 1st one (RDTSC) by the following patch.

All processors that support VMX have that feature, and guests (Xen) depend on
it.  As we already implement it, advertize it to the guest.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/vmx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index cc8ad98..8092f25 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -1990,7 +1990,7 @@ static __init void nested_vmx_setup_ctls_msrs(void)
 #endif
 		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
 		CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
-		CPU_BASED_RDPMC_EXITING |
+		CPU_BASED_RDPMC_EXITING | CPU_BASED_RDTSC_EXITING |
 		CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
 	/*
 	 * We can allow some features even when not supported by the
Comment 1 Nadav Har'El 2013-02-11 15:10:48 UTC
After Avi fixed the missing support CPU_BASED_RDTSC_EXITING, what remains (at least) is missing support for VM_EXIT_ACK_INTR_ON_EXIT.

See the following thread on how this support might be added:
https://patchwork.kernel.org/patch/789632/
Comment 2 Zhou, Chao 2014-05-20 07:33:08 UTC
kvm.git + qemu.git:198c74f4_a41b2c99
host kernel :3.15.0-rc1
test on Romley_EP, crete L1(xen on kvm) guest with "-cpu host", then create L2 guest, L1 guest can boot up.

some log with command "xl dmesg" in L1 guest
(XEN) Platform timer is 100.000MHz HPET
(XEN) Allocated console ring of 32 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
Comment 3 Paolo Bonzini 2014-05-20 11:02:40 UTC
Patches were committed to kvm/next and will be included in 3.16.