Created attachment 282583 [details] libvirt XML I'm not sure if it is a supported scenario to run Hyper-V inside KVM, however this worked for me on Intel platform, and I only have this issue on AMD Ryzen. After enabling Hyper-V feature in Windows guest, I could successfully boot into Windows L1 guest desktop, however after that the L1 guest system consumes all available CPU cores, freezes and becomes unusable. Hardware platform: CPU: AMD Ryzen Threadripper 2950X Board: ASUS Prime X399-A (SVM, IOMMU related settings enabled in BIOS) Linux boot command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-0.bpo.4-amd64 root=UUID=<guid> ro quiet apparmor=0 amd_iommu=pt nopti noibrs noibpb nospectre_v2 nospec_store_bypass_disable pcie_aspm=off apparmor=0 KVM module parameters: options kvm ignore_msrs=1 options kvm report_ignored_msrs=0 options kvm allow_unsafe_assigned_interrupts=1 options vfio_iommu_type1 allow_unsafe_interrupts=1 QEMU: QEMU emulator version 4.0.0 built with ./configure --target-list=x86_64-softmmu --audio-drv-list=pa,alsa,sdl,oss --enable-attr --enable-bluez --enable-brlapi --enable-virtfs --enable-cap-ng --enable-curl --enable-fdt --enable-gnutls --disable-gtk --disable-vte --enable-libiscsi --enable-libnfs --enable-curses --enable-numa --enable-rbd --enable-glusterfs --enable-vnc-sasl --enable-sdl --enable-seccomp --enable-spice --enable-libusb --enable-usb-redir --enable-libssh2 --enable-vde --enable-xfsctl --enable-vnc --enable-vnc-jpeg --enable-vnc-png --enable-kvm --enable-vhost-net --enable-opengl --enable-virglrenderer --enable-avx2 --enable-tpm --enable-vhost-kernel --enable-virtfs libvirt XML is attached L1 Guest OS: Windows Server 2016, 2019 and Windows 10 L2 Guest OS: (Not ever had a chance to start one) `perf kvm stat live` shows unusual numbers of vmrun and msr, comparing to normal VMs: VM-EXIT Samples Samples% Time% Min Time Max Time Avg time vmrun 71298 47.13% 2.61% 1.07us 46197.83us 19.09us ( +- 9.11% ) msr 71217 47.08% 1.24% 0.61us 31213.68us 9.04us ( +- 10.45% ) hlt 3465 2.29% 95.58% 0.70us 78318.83us 14370.28us ( +- 0.93% ) npf 2053 1.36% 0.38% 0.64us 15532.20us 96.25us ( +- 26.48% ) invlpga 1514 1.00% 0.00% 0.28us 60.25us 1.06us ( +- 10.58% ) interrupt 740 0.49% 0.04% 0.22us 15289.36us 25.67us ( +- 80.62% ) vintr 328 0.22% 0.12% 0.37us 31341.32us 194.65us ( +- 59.60% ) stgi 134 0.09% 0.00% 0.32us 49.18us 1.34us ( +- 31.81% ) iret 133 0.09% 0.00% 0.28us 1.52us 0.58us ( +- 3.36% ) io 119 0.08% 0.00% 2.37us 51.02us 14.38us ( +- 4.98% ) hypercall 104 0.07% 0.03% 0.77us 15522.73us 152.12us ( +- 98.10% ) nmi 96 0.06% 0.00% 0.67us 40.84us 2.89us ( +- 20.49% ) write_cr8 66 0.04% 0.00% 0.72us 3.09us 1.77us ( +- 3.19% ) There are also a few number of errors in kmsg after L1 guest boots: [755580.533587] svm_set_msr: 2 callbacks suppressed [755580.533588] SVM: kvm [14227]: vcpu0, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.191889] SVM: kvm [14227]: vcpu1, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.323561] SVM: kvm [14227]: vcpu2, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.482291] SVM: kvm [14227]: vcpu3, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.642842] SVM: kvm [14227]: vcpu4, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.803020] SVM: kvm [14227]: vcpu5, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755581.963498] SVM: kvm [14227]: vcpu6, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755582.123565] SVM: kvm [14227]: vcpu7, guest rIP: 0xfffff986014dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755616.107080] SVM: kvm [14227]: vcpu0, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755616.778183] SVM: kvm [14227]: vcpu1, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755616.910078] SVM: kvm [14227]: vcpu2, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755617.047056] SVM: kvm [14227]: vcpu3, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755617.180127] SVM: kvm [14227]: vcpu4, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755617.327040] SVM: kvm [14227]: vcpu5, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755617.487152] SVM: kvm [14227]: vcpu6, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0 [755617.626337] SVM: kvm [14227]: vcpu7, guest rIP: 0xfffffb735a8dca0c unimplemented wrmsr: 0xc0010115 data 0x0
I am having a similar probably the same issue with proxmox 6/windows server 2019. i am able to boot an ubuntu image as l2 guest. But Either the l1 system is unresponsive or it crashes with a bluesceen before reaching the gui. Bluescreens seem to be all watchdog related but vary. (DPC Watchdog Violation,clock watchdog timeout). Hardware platform: CPU: AMD Ryzen Threadripper 2950X Board: ASRock X399 Taichi X399-A (SVM, IOMMU related settings enabled in BIOS) root@pve-master:~# grep -H '' /sys/module/kvm_amd/parameters/* /sys/module/kvm_amd/parameters/avic:0 /sys/module/kvm_amd/parameters/nested:1 /sys/module/kvm_amd/parameters/npt:1 /sys/module/kvm_amd/parameters/pause_filter_count:3000 /sys/module/kvm_amd/parameters/pause_filter_count_grow:2 /sys/module/kvm_amd/parameters/pause_filter_count_max:65535 /sys/module/kvm_amd/parameters/pause_filter_count_shrink:0 /sys/module/kvm_amd/parameters/pause_filter_thresh:128 /sys/module/kvm_amd/parameters/sev:0 /sys/module/kvm_amd/parameters/vgif:1 /sys/module/kvm_amd/parameters/vls:1 root@pve-master:~# grep -H '' /sys/module/kvm/parameters/* /sys/module/kvm/parameters/enable_vmware_backdoor:N /sys/module/kvm/parameters/force_emulation_prefix:N /sys/module/kvm/parameters/halt_poll_ns:200000 /sys/module/kvm/parameters/halt_poll_ns_grow:0 /sys/module/kvm/parametQers/halt_poll_ns_shrink:0 /sys/module/kvm/parameters/ignore_msrs:Y /sys/module/kvm/parameters/kvmclock_periodic_sync:Y /sys/module/kvm/parameters/lapic_timer_advance_ns:4294967295 /sys/module/kvm/parameters/min_timer_period_us:200 /sys/module/kvm/parameters/report_ignored_msrs:N /sys/module/kvm/parameters/tsc_tolerance_ppm:250 /sys/module/kvm/parameters/vector_hashing:Y /usr/bin/kvm -id 114 -name server2019 -chardev socket,id=qmp,path=/var/run/qemu-server/114.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/114.pid -daemonize -smbios type=1,uuid=8167be1e-8f3d-4667-8535-62a035ebd087 -smp 6,sockets=1,cores=6,maxcpus=6 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/114.vnc,password -no-hpet -cpu EPYC,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,enforce,vendor=AuthenticAMD -m 8192 -device vmgenid,guid=53ded4c7-f067-4ae6-8ab0-6a571ccbc1ba -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:6ff4f776953c -drive file=/pool_t1/t1_dir//template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive file=/pool_t1/t1_dir//template/iso/winserver19_17763.1.180914-1434.rs5_release_SERVER_EVAL_x64fre_de-de.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201 -drive file=/dev/zvol/pool_t1/vmdata/vm-114-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap114i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=16:72:12:9F:33:FA,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard -cpu host,+svm
I have same errors pop in "dmesg", but only when my second VM (Ubuntu 18.04) is running. Windows 10 VM does not seem to be causing these (at least not for me). Kernel 5.1.16-050116-generic (Elementary OS 5), QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.15). Can post configs later if needed
I have the exact same errors, but here my Windows 10 Pro L1 freezes few seconds start. I'm trying to use Windows SandBox. Ryzen 3700x // Gigabyte X570 Aorus Elite Kernel 5.3.0. I have AMD SVM enabled in BIOS and IOMMU disabled. beast ~ # grep -H '' /sys/module/kvm_amd/parameters/* /sys/module/kvm_amd/parameters/avic:0 /sys/module/kvm_amd/parameters/dump_invalid_vmcb:N /sys/module/kvm_amd/parameters/nested:1 /sys/module/kvm_amd/parameters/npt:1 /sys/module/kvm_amd/parameters/nrips:1 /sys/module/kvm_amd/parameters/pause_filter_count:3000 /sys/module/kvm_amd/parameters/pause_filter_count_grow:2 /sys/module/kvm_amd/parameters/pause_filter_count_max:65535 /sys/module/kvm_amd/parameters/pause_filter_count_shrink:0 /sys/module/kvm_amd/parameters/pause_filter_thresh:128 /sys/module/kvm_amd/parameters/sev:0 /sys/module/kvm_amd/parameters/vgif:1 /sys/module/kvm_amd/parameters/vls:1 beast ~ # grep -H '' /sys/module/kvm/parameters/* /sys/module/kvm/parameters/enable_vmware_backdoor:N /sys/module/kvm/parameters/force_emulation_prefix:N /sys/module/kvm/parameters/halt_poll_ns:200000 /sys/module/kvm/parameters/halt_poll_ns_grow:2 /sys/module/kvm/parameters/halt_poll_ns_grow_start:10000 /sys/module/kvm/parameters/halt_poll_ns_shrink:0 /sys/module/kvm/parameters/ignore_msrs:Y /sys/module/kvm/parameters/kvmclock_periodic_sync:Y /sys/module/kvm/parameters/lapic_timer_advance_ns:-1 /sys/module/kvm/parameters/min_timer_period_us:200 /sys/module/kvm/parameters/pi_inject_timer:0 /sys/module/kvm/parameters/report_ignored_msrs:Y /sys/module/kvm/parameters/tsc_tolerance_ppm:250 /sys/module/kvm/parameters/vector_hashing:Y QEMU Command line: usr/bin/qemu-system-x86_64 -name guest=Win10Devel,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Win10Devel/master-key.aes -machine pc-q35-4.0.1,accel=kvm,usb=off,dump-guest-core=off -cpu EPYC-IBPB,svm=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 6072 -overcommit mem-lock=off -smp 4,sockets=1,cores=2,threads=2 -uuid 466412cf-36da-4297-ba3e-6f7e7b0b0b5d -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=22,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive file=/mnt/storage/Eduardo/VM/KVM/win10_desenvolvimento.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2,write-cache=on -drive if=none,id=drive-sata0-0-1,readonly=on -device ide-cd,bus=ide.1,drive=drive-sata0-0-1,id=sata0-0-1,bootindex=1 -drive if=none,id=drive-fdc0-0-0 -device isa-fdc,driveA=drive-fdc0-0-0 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:2f:ca,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,fd=26,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spiceport,id=charchannel2,name=org.spice-space.webdav.0 -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.spice-space.webdav.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=134217728,vram_size=134217728,vram64_size_mb=0,vgamem_mb=64,max_outputs=1,bus=pcie.0,addr=0x1 -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
I have the same problem on a Ryzen 3950x (Asus PRIME X570-PRO) with kernel 5.6.19 (Fedora 32).
I have the same problem on a Ryzen 3950x, are this a problem in kvm or in windows guest or in qemu?
The command line I am using ``` /usr/bin/kvm -id 101 -name Win10-Video -chardev socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/101.pid -daemonize -smbios type=1,uuid=0e1a8f04-2fcd-4e7b-a615-b5f60c17c244 -smp 16,sockets=1,cores=16,maxcpus=16 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/101.vnc,password -no-hpet -cpu kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 16384 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=4d51db1b-14b4-48ee-a10e-87e12165dd90 -device nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device usb-host,bus=xhci.0,hostbus=4,hostport=2,id=usb0 -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:fbd3a8f979d -drive file=/var/lib/vz/template/iso/virtio-win-0.1.189.iso,if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive file=/var/lib/vz/template/iso/cn_windows_10_business_editions_version_2004_updated_sep_2020_x64_dvd_7134ba4b.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201 -drive file=/dev/pve/vm-101-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=AE:79:28:76:97:65,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35+pve0 -global kvm-pit.lost_tick_policy=discard -cpu host,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep,+svm,-hypervisor ```
(In reply to Yonggang Luo from comment #6) >-cpu > host,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff, > hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt, > +lahf_lm,+sep,+svm,-hypervisor > ``` Try adding 'hv_stimer_direct' to the list, Hyper-V can't use synthetic timers otherwise. Also, why do you need '-hypervisor' flag? Could you try without it? Also, please try with the latest upstream kernel (5.9). 5.3 has a lot of known nested SVM related bugs.
An upgrade to kernel 5.8.13 solved the issue for me.
(In reply to Timo Sandmann from comment #8) > An upgrade to kernel 5.8.13 solved the issue for me. What's youd qemu command line parameters?
Confirmed that it is no longer an issue on 6.1.0 kernel.