Reported also here: https://sourceforge.net/p/kvm/bugs/556/ Nested Virtualization summary: ____________________ | | | Vbox Guest | | | ----------------| | |Guest OS: Vbox | Vbox in KVM => KO! :-( | ----------------- | | Host OS: kvm | -------------------- ___________________ | | | Vbox Guest | | | ------------- -| | |Guest OS: Vbox | Vbox in Vbox => OK!! | ---------------- | | Host OS: Vbox | ------------------- ---------------- Host OS details ----------------- Host cpu model: model name: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz Host cpu flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap Host OS: cat /etc/issue Ubuntu 14.04.5 LTS Host kvm version: kvm -version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.30) Host kvm options enabled: cat /sys/module/kvm_intel/parameters/nested Y Host kernel version and arch: uname -a Linux xxx 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Qemu command line you are using to start the Guest OS, where Virtualbox is installed: Actually I use virt-manager with the following settings CPU Model: Haswell + Copy host CPU configuration ---------------- Guest OS details----------------- Guest OS: lsb_release -da Debian GNU/Linux 8.6 (jessie) Guest kernel version and arch: uname -a Linux yyy 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 x86_64 Virtualbox version: 5.1.8 r111374 Vbox options: Paravirtualization Interface ANY, Enable VT-x/AMD-V, Enable Nested Paging ---------------- Vbox Guest OS details ----------------- Vbox Guest OS: Debian GNU/Linux 8 Vbox Guest kernel and version: uname -a Linux zzz 4.6.0-0-.bpo.1-686 #1 SMP Debian 4.6.4-1~bpo8+1 i686 GNU/Linux Vbox Guest log: cat Vbox.log VirtualBox VM 5.1.8 r111374 linux.amd64 (Oct 18 2016 15:36:00) release log 00:00:00.318696 Log opened 2016-11-19T10:35:51.837374000Z 00:00:00.318698 Build Type: release 00:00:00.318700 OS Product: Linux 00:00:00.318700 OS Release: 3.16.0-4-amd64 00:00:00.318701 OS Version: #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) 00:00:00.318718 DMI Product Name: Standard PC (i440FX + PIIX, 1996) 00:00:00.318723 DMI Product Version: pc-i440fx-trusty 00:00:00.318777 Host RAM: 7813MB (7.6GB) total, 7504MB (7.3GB) available 00:00:00.318781 Executable: /usr/lib/virtualbox/VirtualBox 00:00:00.318782 Process ID: 12030 00:00:00.318782 Package type: LINUX_64BITS_DEBIAN_8_0 00:00:00.345315 Installed Extension Packs: 00:00:00.345341 None installed! 00:00:00.346431 Console: Machine state changed to 'Starting' 00:00:00.398176 Qt version: 5.3.2 00:00:00.404989 GUI: UIMediumEnumerator: Medium-enumeration finished! 00:00:00.405720 X Server details: vendor: The X.Org Foundation, release: 11604000, protocol version: 11.0, display string: :0.0 00:00:00.405734 Using XKB for keycode to scan code conversion Guest OS log: dmesg [ 1645.678775] SUPR0GipMap: fGetGipCpu=0xb [ 1646.128514] general protection fault: 0000 [#1] SMP [ 1646.128521] Modules linked in: ipt_MASQUERADE xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack bridge stp llc aufs(C) ppdev pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) joydev kvm_intel kvm crc32_pclmul evdev tun virtio_balloon serio_raw pcspkr aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd virtio_console parport_pc parport pvpanic snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore ttm processor thermal_sys i2c_piix4 drm_kms_helper drm button i2c_core fuse autofs4 hid_generic usbhid hid ext4 crc16 mbcache jbd2 ata_generic virtio_blk virtio_net crct10dif_pclmul crct10dif_common crc32c_intel floppy psmouse [ 1646.128571] ata_piix uhci_hcd ehci_hcd virtio_pci virtio_ring virtio libata scsi_mod usbcore usb_common [ 1646.128586] CPU: 0 PID: 12061 Comm: EMT Tainted: G C O 3.16.0-4-amd64 #1 Debian 3.16.36-1+deb8u2 [ 1646.128589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [ 1646.128592] task: ffff88022767a190 ti: ffff8800ba174000 task.ti: ffff8800ba174000 [ 1646.128594] RIP: 0010:[<ffffffffa058faf1>] [<ffffffffa058faf1>] 0xffffffffa058faf1 [ 1646.128606] RSP: 0018:ffff8800ba177dc0 EFLAGS: 00000206 [ 1646.128608] RAX: 00000000001406f0 RBX: 00000000ffffffdb RCX: 000000000000009b [ 1646.128610] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800ba177d20 [ 1646.128612] RBP: ffff8800ba177df0 R08: ffffffff8160da48 R09: 00000000756e6547 [ 1646.128613] R10: 000000006c65746e R11: 0000000049656e69 R12: 000000000f8bfbff [ 1646.128615] R13: 0000000000000020 R14: ffff88022a006890 R15: 0000000000000000 [ 1646.128618] FS: 00007f3fb9615700(0000) GS:ffff880233c00000(0000) knlGS:0000000000000000 [ 1646.128621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1646.128623] CR2: 00007f3fb939c000 CR3: 000000022a377000 CR4: 00000000001406f0 [ 1646.128634] Stack: [ 1646.128636] ffff8800ba177df0 ffffffff00000000 0000000000000000 0000000000000000 [ 1646.128640] 0000000000000000 ffffc90000f6e010 ffff8800ba177e10 ffffffffa05a9b63 [ 1646.128643] 0000000000000000 ffffffffa04721a0 ffff8800ba177e88 ffffffffa04375b0 [ 1646.128649] Call Trace: [ 1646.128667] [<ffffffffa04375b0>] ? supdrvIOCtl+0x1fc0/0x3400 [vboxdrv] [ 1646.128675] [<ffffffffa0431571>] ? VBoxDrvLinuxIOCtl_5_1_8+0x121/0x210 [vboxdrv] [ 1646.128697] [<ffffffff811bce4f>] ? do_vfs_ioctl+0x2cf/0x4b0 [ 1646.128701] [<ffffffff811bd0b1>] ? SyS_ioctl+0x81/0xa0 [ 1646.128714] [<ffffffff8151a5d8>] ? async_page_fault+0x28/0x30 [ 1646.128719] [<ffffffff8151858d>] ? system_call_fast_compare_end+0x10/0x15 [ 1646.128721] Code: 85 c0 0f 88 68 fc ff ff b9 3a 00 00 00 0f 32 48 c1 e2 20 89 c0 48 09 d0 48 89 05 ab 7b 0f 00 0f 20 e0 48 89 05 89 7b 0f 00 b1 9b <0f> 32 48 c1 e2 20 89 c0 b9 80 00 00 c0 48 09 d0 48 89 05 80 7b [ 1646.128753] RIP [<ffffffffa058faf1>] 0xffffffffa058faf1 [ 1646.128761] RSP <ffff8800ba177dc0> [ 1646.128777] ---[ end trace d7e0c42d0708c3d2 ]---
I guess this isn't fixed, even in mainline. I've just checked and it seems that this is 100% reproducible. L1: KVM * (Linux 4.10-rc4) L2: VirtualBox /Vmware (Linux 4.9.4) * dmesg when runninng virtualbox [ 40.583722] SUPR0GipMap: fGetGipCpu=0xb [ 41.047407] general protection fault: 0000 [#1] SMP [ 41.047410] Modules linked in: nls_utf8 udf crc_itu_t fuse joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_broute bridge stp llc ebtable_nat ip6table_raw ip6table_security ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_raw iptable_security iptable_mangle iptable_nat nf_conntrack_ipv4 vboxpci(OE) nf_defrag_ipv4 vboxnetadp(OE) nf_nat_ipv4 nf_nat vboxnetflt(OE) nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables vboxdrv(OE) kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev virtio_balloon qemu_fw_cfg parport_pc parport acpi_cpufreq tpm_tis tpm_tis_core tpm i2c_piix4 nfsd auth_rpcgss [ 41.047423] nfs_acl lockd grace sunrpc virtio_net virtio_blk virtio_console qxl drm_kms_helper ttm drm virtio_pci crc32c_intel serio_raw virtio_ring virtio ata_generic pata_acpi [ 41.047427] CPU: 1 PID: 2191 Comm: EMT Tainted: G OE 4.9.3-200.fc25.x86_64 #1 [ 41.047428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-20161122_114906-anatol 04/01/2014 [ 41.047429] task: ffff9d56e4fdbe00 task.stack: ffffc0ab427a8000 [ 41.047429] RIP: 0010:[<ffffffffc000baa7>] [<ffffffffc000baa7>] 0xffffffffc000baa7 [ 41.047431] RSP: 0018:ffffc0ab427abd58 EFLAGS: 00050206 [ 41.047432] RAX: 00000000003406e0 RBX: 00000000ffffffdb RCX: 000000000000009b [ 41.047432] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffc0ab427abcb0 [ 41.047587] RBP: ffffc0ab427abd78 R08: 0000000000000004 R09: 00000000003406e0 [ 41.047588] R10: 0000000049656e69 R11: 000000000f8bfbff R12: 0000000000000020 [ 41.047588] R13: 0000000000000000 R14: ffffc0ab4800107c R15: ffffffffc04922a0 [ 41.047589] FS: 00007f27613cb700(0000) GS:ffff9d577fd00000(0000) knlGS:0000000000000000 [ 41.047590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 41.047591] CR2: 00007f2761158000 CR3: 0000000138bf0000 CR4: 00000000003406e0 [ 41.047592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 41.047593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 41.047593] Stack: [ 41.047594] 0000000000000000 ffffffff00000000 0000000000000000 0000000000000002 [ 41.047595] ffffc0ab427abd98 ffffffffc0026a23 ffffc0ab48001010 ffff9d5779db1a90 [ 41.047597] ffffc0ab427abe18 ffffffffc0457420 ffffc0ab427abdf8 0000000000040296 [ 41.047598] Call Trace: [ 41.047606] [<ffffffffc0457420>] ? supdrvIOCtl+0x2dc0/0x32c0 [vboxdrv] [ 41.047609] [<ffffffffc04505e0>] ? VBoxDrvLinuxIOCtl_5_1_14+0x150/0x250 [vboxdrv] [ 41.047612] [<ffffffff9f26db43>] ? do_vfs_ioctl+0xa3/0x5f0 [ 41.047613] [<ffffffff9f06280b>] ? __do_page_fault+0x23b/0x4e0 [ 41.047614] [<ffffffff9f26e109>] ? SyS_ioctl+0x79/0x90 [ 41.047616] [<ffffffff9f81bbf7>] ? entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 41.047617] Code: 88 d1 fc ff ff b9 3a 00 00 00 0f 32 48 c1 e2 20 89 c0 48 09 d0 48 89 05 d8 4b 0f 00 0f 20 e0 b9 9b 00 00 00 48 89 05 b1 4b 0f 00 <0f> 32 48 c1 e2 20 89 c0 b9 80 00 00 c0 48 09 d0 48 89 05 aa 4b [ 41.047629] RIP [<ffffffffc000baa7>] 0xffffffffc000baa7 [ 41.047630] RSP <ffffc0ab427abd58> [ 41.047631] ---[ end trace 2d3de5d7dc5b188a ]--- Is there any news regards supporting other hypervisors except kvm and hyper-v? Could someone explain why this "support" is needed? I mean since kvm is just working fine.
There is also a PR opened on Virtualbox bug tracking system [https://www.virtualbox.org/ticket/14965], so it is not a limited concern.
@micio Thank you for mentioning this. According to frank's last message it seems that this isn't kvm's fault at all? He says that qemu doesn't implement MSR 0x9B (IA32_SMM_MONITOR_CTL). Do you know if a bug report already exists for qemu?
A quick look into linux made me realize that it's not qemu's fault. According to the comments in the kvm code, it seems that it's simply not supported right now: https://github.com/torvalds/linux/blob/master/arch/x86/kvm/vmx.c#L9874 https://github.com/torvalds/linux/blob/master/arch/x86/kvm/vmx.c#L9883
@Paul: thanks a lot for the update Is there any chance to add such a support?
I answered on the VirtualBox ticket. The SDM is ambiguous, but there's no reason for KVM to implement this MSR and some lines in the SDM back this choice. Do you have any pointer to VirtualBox OSE code that reads the MSR? It makes little sense for vbox to read it, too.