1. Please describe the problem: Starting with kernel 5.1.0-rc6, kvm_intel can no longer be loaded in nested kvm/guests [root@undercloud-0-rhosp10 ~]# modprobe kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error 2. What is the Version-Release number of the kernel: 5.1.0-rc7 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Yes it seems to have appearded between 5.1.0-rc4 (it worked) and 5.1.0-rc6 (it no longer worked) 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes, update to 5.1.0-rc6 and try to modprobe kvm_intel inside a guest where the VMX capabilities has been exposted 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Yes 6. Are you running any modules that not shipped with directly Fedora's kernel?: No 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag.
If I would have to guess, I would blame my own commit: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”) As in kvm_intel’s setup_vmcs_config() it can be seen that CPU_BASED_RDPMC_EXITING is required in order for KVM to load. Therefore, I assume the issue is that now L1 guest is not exposed with CPU_BASED_RDPMC_EXITING. My patch is suppose to hide CPU_BASED_RDPMC_EXITING from L1 only in case L1 vCPU is not exposed with PMU. Can you provide more details on the vCPU your setup expose to L1? Have you explicitly disabled PMU from L1 vCPU? Can you run “cpuid -r” on shell and post here it’s output?
If I would have to guess, I would blame my own commit: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”) As in kvm_intel’s setup_vmcs_config() it can be seen that CPU_BASED_RDPMC_EXITING is required in order for KVM to load. Therefore, I assume the issue is that now L1 guest is not exposed with CPU_BASED_RDPMC_EXITING. My patch is suppose to hide CPU_BASED_RDPMC_EXITING from L1 only in case L1 vCPU is not exposed with PMU. Can you provide more details on the vCPU your setup expose to L1? Have you explicitly disabled PMU from L1 vCPU? Can you run “cpuid -r” on shell and post here it’s output? -Liran > On 7 May 2019, at 23:45, bugzilla-daemon@bugzilla.kernel.org wrote: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=cB5pEya2zKbSTvVpkeIHKYlQ1F9qW__mnLe0hXnlBCM&s=O2zSR2K1fTcD1Ps40Q_i-ZmS9tVsPYjupbQmw-LCMPk&e= > > Bug ID: 203543 > Summary: Starting with kernel 5.1.0-rc6, kvm_intel can no > longer be loaded in nested kvm/guests > Product: Virtualization > Version: unspecified > Kernel Version: 5.1.0-rc6 > Hardware: Intel > OS: Linux > Tree: Mainline > Status: NEW > Severity: blocking > Priority: P1 > Component: kvm > Assignee: virtualization_kvm@kernel-bugs.osdl.org > Reporter: hilld@binarystorm.net > Regression: No > > 1. Please describe the problem: > Starting with kernel 5.1.0-rc6, kvm_intel can no longer be loaded in nested > kvm/guests > > [root@undercloud-0-rhosp10 ~]# modprobe kvm_intel > modprobe: ERROR: could not insert 'kvm_intel': Input/output error > > > 2. What is the Version-Release number of the kernel: > 5.1.0-rc7 > > 3. Did it work previously in Fedora? If so, what kernel version did the issue > *first* appear? Old kernels are available for download at > > https://urldefense.proofpoint.com/v2/url?u=https-3A__koji.fedoraproject.org_koji_packageinfo-3FpackageID-3D8&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=cB5pEya2zKbSTvVpkeIHKYlQ1F9qW__mnLe0hXnlBCM&s=1wtgL9MEqhN6ZwOZRMKlcW6LYP3zCgz4-1lh8aXyTWo&e= > : > Yes it seems to have appearded between 5.1.0-rc4 (it worked) and 5.1.0-rc6 > (it > no longer worked) > > 4. Can you reproduce this issue? If so, please provide the steps to reproduce > the issue below: > Yes, update to 5.1.0-rc6 and try to modprobe kvm_intel inside a guest where > the > VMX capabilities has been exposted > > > 5. Does this problem occur with the latest Rawhide kernel? To install the > Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by > ``sudo dnf update --enablerepo=rawhide kernel``: > Yes > > > 6. Are you running any modules that not shipped with directly Fedora's > kernel?: > No > > 7. Please attach the kernel logs. You can get the complete kernel log > for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the > issue occurred on a previous boot, use the journalctl ``-b`` flag. > > -- > You are receiving this mail because: > You are watching the assignee of the bug.
The guest vmx CPU flag is exposed like this: <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>IvyBridge-IBRS</model> <feature policy='require' name='vmx'/> </cpu> and here the the "cpuid -r" from the guest : CPU 0: 0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000306a9 ebx=0x00000800 ecx=0xffb82223 edx=0x078bfbff 0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001 0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006 0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000 0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000000 0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000000 0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000 0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d 0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800 0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33 0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972 0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000 0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140 0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 CPU 1: 0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000306a9 ebx=0x01000800 ecx=0xffb82223 edx=0x078bfbff 0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001 0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006 0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000 0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000001 0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000001 0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000 0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d 0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800 0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33 0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972 0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000 0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140 0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 CPU 2: 0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000306a9 ebx=0x02000800 ecx=0xffb82223 edx=0x078bfbff 0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001 0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006 0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000 0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000002 0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000002 0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000 0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d 0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800 0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33 0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972 0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000 0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140 0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 CPU 3: 0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000306a9 ebx=0x03000800 ecx=0xffb82223 edx=0x078bfbff 0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001 0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001 0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006 0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000 0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000003 0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000003 0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000 0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d 0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800 0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33 0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972 0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000 0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140 0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000 0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
I didn't do anything fancy... only enabled nested kvm with: [root@wolfe modprobe.d]# cat kvm.conf ### ### This configuration file was provided by the qemu package. ### Feel free to update as needed. ### ### ### Set these options to enable nested virtualization ### #options kvm_intel nested=1 #options kvm_amd nested=1 options kvm_intel nested=1 and exposed the flag in the libvirtd kvm definition. Here is the hardware description of the VM: <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-4.0'>hvm</type> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>IvyBridge-IBRS</model> <feature policy='require' name='vmx'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm>
I can confirm that reverting that commit solves the problem: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”)
Revert sent, the previous KVM behavior is correct. The failing unit test that motivated the change needs to be modified to gracefully handle the #GP or avoid it entirely.
On Wed, May 08, 2019 at 07:00:54PM +0300, Liran Alon wrote: > +Paolo > > What are your thoughts on this? What is the reason that KVM relies on > CPU_BASED_RDPMC_EXITING to be exposed from underlying CPU? How is it critical > for it’s functionality? If it’s because we want to make sure that we hide > host PMCs, we should condition this to be a min requirement of kvm_intel only > in case underlying CPU exposes PMU to begin with. Do you agree? If yes, I > can create the patch to fix this. I sent a revert of the change to hide CPU_BASED_RDPMC_EXITING, KVM's previous behavior is correct. The RDPMC instruction was introduced long before Architctural Perf Mon and so the existence of the exiting control is dependent only on the instruction, e.g. P4 (Prescott), Core (Yonah) and Core2 (Merom) all support VMX and RDPMC with non-archictectural perf mon capabilities. The KVM unit test first execute RDPMC with interception disabled in the unit test host, i.e. the #GP is the correct architectural behavior and needs to be handled by the unit test. The most robust fix would be to eat any #GP on RDPMC in the unit test, though it's likely much simpler to only execute RDPMC with interception disabled if arch perf mon is supported.
+Paolo What are your thoughts on this? What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed from underlying CPU? How is it critical for it’s functionality? If it’s because we want to make sure that we hide host PMCs, we should condition this to be a min requirement of kvm_intel only in case underlying CPU exposes PMU to begin with. Do you agree? If yes, I can create the patch to fix this. -Liran > On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e= > > --- Comment #5 from David Hill (hilld@binarystorm.net) --- > I can confirm that reverting that commit solves the problem: > > e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”) > > -- > You are receiving this mail because: > You are watching the assignee of the bug.
Hi guys, I justed tested kernel 5.2.1-rc1 which contains the following commit reverting the previously mentionned commit but... the problem is still there: [root@wolfe linux-stable]# su - jenkins -s /bin/bash [jenkins@wolfe ~]$ ssh root@192.168.122.2 Activate the web console with: systemctl enable --now cockpit.socket Last login: Tue May 21 08:53:28 2019 from 192.168.122.1 [root@undercloud-0-rhosp15 ~]# modprobe kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error [root@undercloud-0-rhosp15 ~]# logout Connection to 192.168.122.2 closed. [jenkins@wolfe ~]$ uname -a Linux wolfe.orion 5.2.0-20190521081919+ #6 SMP Tue May 21 08:25:14 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux I'll try reverting the commit and then reverting only the commit that causes issues. commit f93f7ede087f2edcc18e4b02310df5749a6b5a61 Author: Sean Christopherson <sean.j.christopherson@intel.com> Date: Wed May 8 09:08:19 2019 -0700 Revert "KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU" The RDPMC-exiting control is dependent on the existence of the RDPMC instruction itself, i.e. is not tied to the "Architectural Performance Monitoring" feature. For all intents and purposes, the control exists on all CPUs with VMX support since RDPMC also exists on all VCPUs with VMX supported. Per Intel's SDM: The RDPMC instruction was introduced into the IA-32 Architecture in the Pentium Pro processor and the Pentium processor with MMX technology. The earlier Pentium processors have performance-monitoring counters, but they must be read with the RDMSR instruction. Because RDPMC-exiting always exists, KVM requires the control and refuses to load if it's not available. As a result, hiding the PMU from a guest breaks nested virtualization if the guest attemts to use KVM. While it's not explicitly stated in the RDPMC pseudocode, the VM-Exit check for RDPMC-exiting follows standard fault vs. VM-Exit prioritization for privileged instructions, e.g. occurs after the CPL/CR0.PE/CR4.PCE checks, but before the counter referenced in ECX is checked for validity. In other words, the original KVM behavior of injecting a #GP was correct, and the KVM unit test needs to be adjusted accordingly, e.g. eat the #GP when the unit test guest (L3 in this case) executes RDPMC without RDPMC-exiting set in the unit test host (L2). This reverts commit e51bfdb68725dc052d16241ace40ea3140f938aa. Fixes: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU") Reported-by: David Hill <hilld@binarystorm.net> Cc: Saar Amar <saaramar@microsoft.com> Cc: Mihai Carabas <mihai.carabas@oracle.com> Cc: Jim Mattson <jmattson@google.com> Cc: Liran Alon <liran.alon@oracle.com> Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> On 2019-05-08 6:14 p.m., bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=203543 > > --- Comment #8 from Liran Alon (liran.alon@oracle.com) --- > +Paolo > > What are your thoughts on this? > What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed > from underlying CPU? How is it critical for it’s functionality? > If it’s because we want to make sure that we hide host PMCs, we should > condition this to be a min requirement of kvm_intel only in case underlying > CPU > exposes PMU to begin with. > Do you agree? If yes, I can create the patch to fix this. > > -Liran > >> On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote: >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e= >> >> --- Comment #5 from David Hill (hilld@binarystorm.net) --- >> I can confirm that reverting that commit solves the problem: >> >> e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports >> PMU”) >> >> -- >> You are receiving this mail because: >> You are watching the assignee of the bug.
Reverting both commits solves this problem: f93f7ede087f2edcc18e4b02310df5749a6b5a61 e51bfdb68725dc052d16241ace40ea3140f938aa On 2019-05-21 8:57 a.m., bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=203543 > > --- Comment #9 from dhill@redhat.com --- > Hi guys, > > I justed tested kernel 5.2.1-rc1 which contains the following commit > reverting the previously mentionned commit but... the problem is still > there: > > > [root@wolfe linux-stable]# su - jenkins -s /bin/bash > [jenkins@wolfe ~]$ ssh root@192.168.122.2 > Activate the web console with: systemctl enable --now cockpit.socket > > Last login: Tue May 21 08:53:28 2019 from 192.168.122.1 > [root@undercloud-0-rhosp15 ~]# modprobe kvm_intel > modprobe: ERROR: could not insert 'kvm_intel': Input/output error > [root@undercloud-0-rhosp15 ~]# logout > Connection to 192.168.122.2 closed. > [jenkins@wolfe ~]$ uname -a > Linux wolfe.orion 5.2.0-20190521081919+ #6 SMP Tue May 21 08:25:14 EDT > 2019 x86_64 x86_64 x86_64 GNU/Linux > > > I'll try reverting the commit and then reverting only the commit that > causes issues. > > commit f93f7ede087f2edcc18e4b02310df5749a6b5a61 > Author: Sean Christopherson <sean.j.christopherson@intel.com> > Date: Wed May 8 09:08:19 2019 -0700 > > Revert "KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU" > > The RDPMC-exiting control is dependent on the existence of the RDPMC > instruction itself, i.e. is not tied to the "Architectural Performance > Monitoring" feature. For all intents and purposes, the control exists > on all CPUs with VMX support since RDPMC also exists on all VCPUs with > VMX supported. Per Intel's SDM: > > The RDPMC instruction was introduced into the IA-32 Architecture in > the Pentium Pro processor and the Pentium processor with MMX > technology. > The earlier Pentium processors have performance-monitoring > counters, but > they must be read with the RDMSR instruction. > > Because RDPMC-exiting always exists, KVM requires the control and > refuses > to load if it's not available. As a result, hiding the PMU from a > guest > breaks nested virtualization if the guest attemts to use KVM. > > While it's not explicitly stated in the RDPMC pseudocode, the VM-Exit > check for RDPMC-exiting follows standard fault vs. VM-Exit > prioritization > for privileged instructions, e.g. occurs after the CPL/CR0.PE/CR4.PCE > checks, but before the counter referenced in ECX is checked for > validity. > > In other words, the original KVM behavior of injecting a #GP was > correct, > and the KVM unit test needs to be adjusted accordingly, e.g. eat > the #GP > when the unit test guest (L3 in this case) executes RDPMC without > RDPMC-exiting set in the unit test host (L2). > > This reverts commit e51bfdb68725dc052d16241ace40ea3140f938aa. > > Fixes: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when > guest supports PMU") > Reported-by: David Hill <hilld@binarystorm.net> > Cc: Saar Amar <saaramar@microsoft.com> > Cc: Mihai Carabas <mihai.carabas@oracle.com> > Cc: Jim Mattson <jmattson@google.com> > Cc: Liran Alon <liran.alon@oracle.com> > Cc: stable@vger.kernel.org > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > > > On 2019-05-08 6:14 p.m., bugzilla-daemon@bugzilla.kernel.org wrote: >> https://bugzilla.kernel.org/show_bug.cgi?id=203543 >> >> --- Comment #8 from Liran Alon (liran.alon@oracle.com) --- >> +Paolo >> >> What are your thoughts on this? >> What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed >> from underlying CPU? How is it critical for it’s functionality? >> If it’s because we want to make sure that we hide host PMCs, we should >> condition this to be a min requirement of kvm_intel only in case underlying >> CPU >> exposes PMU to begin with. >> Do you agree? If yes, I can create the patch to fix this. >> >> -Liran >> >>> On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote: >>> >>> >>> >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e= >>> >>> --- Comment #5 from David Hill (hilld@binarystorm.net) --- >>> I can confirm that reverting that commit solves the problem: >>> >>> e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports >>> PMU”) >>> >>> -- >>> You are receiving this mail because: >>> You are watching the assignee of the bug.
On Tue, May 21, 2019 at 07:11:01AM -0700, Sean Christopherson wrote: > On Tue, May 21, 2019 at 01:37:42PM +0000, bugzilla-daemon@bugzilla.kernel.org > wrote: > > https://bugzilla.kernel.org/show_bug.cgi?id=203543 > > > > --- Comment #10 from moi@davidchill.ca --- > > Reverting both commits solves this problem: > > > > f93f7ede087f2edcc18e4b02310df5749a6b5a61 > > e51bfdb68725dc052d16241ace40ea3140f938aa > > Hmm, that makes no sense, f93f7ede087f is a straight revert of > e51bfdb68725. I do see the same behavior on v5.2-rc1 where hiding the > pmu from L1 breaks nested virtualization, but manually reverting both > commits doesn't change that for me, i.e. there's another bug lurking, > which I'll start hunting. Scratch that, had a brain fart and tested the wrong kernel. I do *NOT* see breakage on v5.2-rc1, at least when running v5.2-rc1 as L1 and probing KVM in L2. When running v5.2-rc1 as L0, what are the values of MSRs 0x482 and 0x48e in L1? > > Any chance the successful test used a different command line or something?
On Tue, May 21, 2019 at 01:37:42PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=203543 > > --- Comment #10 from moi@davidchill.ca --- > Reverting both commits solves this problem: > > f93f7ede087f2edcc18e4b02310df5749a6b5a61 > e51bfdb68725dc052d16241ace40ea3140f938aa Hmm, that makes no sense, f93f7ede087f is a straight revert of e51bfdb68725. I do see the same behavior on v5.2-rc1 where hiding the pmu from L1 breaks nested virtualization, but manually reverting both commits doesn't change that for me, i.e. there's another bug lurking, which I'll start hunting. Any chance the successful test used a different command line or something?
Well, I did this: 1) git checkout v5.2-rc1 2) compile it 3) configure grub to use it 4) reboot 5) boot the existing VM 6) lsmod | grep kvm_intel and at 5) it wasn't loaded. next I did: 1) git checkout v5.2-rc1 2) git revert f93f7ede087f2edcc18e4b02310df5749a6b5a61 3) git revert e51bfdb68725dc052d16241ace40ea3140f938aa. 4) compile that 5) configure grub to use it 6) reboot 7) boot the existing VM 8) lsmod | grep kvm_intel and at 8) it was loaded.
I've verified reverting f93f7ede087f and e51bfdb68725 yields the exact same code as v5.2-rc1. Can you please sanity check that v5.2-rc1 is indeed broken? E.g. ensure there are no modified files when compiling and include the git commit id in the kernel name.
You are right actually. I did a mistake in my code and it didn't pull v5.2-rc1 ... I was still on v5.1.x tag so that's why it didn't work. I guess this is issue is solved and if I hit any bugs with v5.2-rc1 it'd be a new issue since we reverted that commit. On 2019-05-21 2:06 p.m., bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=203543 > > --- Comment #14 from Sean Christopherson (sean.j.christopherson@intel.com) > --- > I've verified reverting f93f7ede087f and e51bfdb68725 yields the exact same > code as v5.2-rc1. Can you please sanity check that v5.2-rc1 is indeed > broken? > E.g. ensure there are no modified files when compiling and include the git > commit id in the kernel name. >
Looks like this issue is back : [root@wolfe linux-stable]# uname -a Linux wolfe.orion 5.4.0-2.fc32.x86_64 #1 SMP Mon Nov 25 22:45:19 UTC 2019 x86_64 [root@undercloud-0-rhosp13 ~]# modprobe kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error
Should I open a new bug for this issue ?
v5.3.2 appears to be working
v5.3.14 appears to be working too ... if I login another VM, I do get the same error message: [root@overcloud-compute-0 ~]# modprobe kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error (undercloud) [stack@undercloud-0-rhosp13 ~]$ sudo lsmod | grep kvm kvm_intel 188688 0 kvm 636931 1 kvm_intel irqbypass 13503 1 kvm
I got an intermittent [root@undercloud-0-rhosp13 ~]# modprobe kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error even with 5.1.0-rc4. This is getting stranger.
Something is constant here and it's that if I was able to load kvm_intel on one of the VMs, no other VMs can modprobe kvm_intel.
Ok with latest kernel and libvirt-5.9.0-1.fc32.x86_64, nested virt is broken but if I downgrade to libvirt-5.6.0-4.fc31.x86_64, problem is solved. It looks like nested virt is broken in libvirt-5.9.0-1. Sorry for the noise. I'll close this bug.