Bug 203543

Summary:	Starting with kernel 5.1.0-rc6, kvm_intel can no longer be loaded in nested kvm/guests
Product:	Virtualization	Reporter:	David Hill (hilld)
Component:	kvm	Assignee:	virtualization_kvm
Status:	RESOLVED CODE_FIX
Severity:	blocking	CC:	liran.alon, seanjc
Priority:	P1
Hardware:	Intel
OS:	Linux
Kernel Version:	5.1.0-rc6	Subsystem:
Regression:	No	Bisected commit-id:

Description David Hill 2019-05-07 20:45:45 UTC

1. Please describe the problem:
Starting with kernel 5.1.0-rc6,  kvm_intel can no longer be loaded in nested kvm/guests

[root@undercloud-0-rhosp10 ~]# modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error


2. What is the Version-Release number of the kernel:
5.1.0-rc7

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
Yes it seems to have appearded between 5.1.0-rc4 (it worked) and 5.1.0-rc6 (it no longer worked)

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
Yes, update to 5.1.0-rc6 and try to modprobe kvm_intel inside a guest where the VMX capabilities has been exposted


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
Yes


6. Are you running any modules that not shipped with directly Fedora's kernel?:
No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Comment 1 Liran Alon 2019-05-07 23:29:40 UTC

If I would have to guess, I would blame my own commit:
e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”)

As in kvm_intel’s setup_vmcs_config() it can be seen that CPU_BASED_RDPMC_EXITING is required in order for KVM to load.
Therefore, I assume the issue is that now L1 guest is not exposed with CPU_BASED_RDPMC_EXITING.

My patch is suppose to hide CPU_BASED_RDPMC_EXITING from L1 only in case L1 vCPU is not exposed with PMU.

Can you provide more details on the vCPU your setup expose to L1?
Have you explicitly disabled PMU from L1 vCPU?
Can you run “cpuid -r” on shell and post here it’s output?

Comment 2 Liran Alon 2019-05-08 04:08:10 UTC

If I would have to guess, I would blame my own commit:
e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”)

As in kvm_intel’s setup_vmcs_config() it can be seen that CPU_BASED_RDPMC_EXITING is required in order for KVM to load.
Therefore, I assume the issue is that now L1 guest is not exposed with CPU_BASED_RDPMC_EXITING.

My patch is suppose to hide CPU_BASED_RDPMC_EXITING from L1 only in case L1 vCPU is not exposed with PMU.

Can you provide more details on the vCPU your setup expose to L1?
Have you explicitly disabled PMU from L1 vCPU?
Can you run “cpuid -r” on shell and post here it’s output?

-Liran

> On 7 May 2019, at 23:45, bugzilla-daemon@bugzilla.kernel.org wrote:
> 
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=cB5pEya2zKbSTvVpkeIHKYlQ1F9qW__mnLe0hXnlBCM&s=O2zSR2K1fTcD1Ps40Q_i-ZmS9tVsPYjupbQmw-LCMPk&e=
> 
>            Bug ID: 203543
>           Summary: Starting with kernel 5.1.0-rc6,  kvm_intel can no
>                    longer be loaded in nested kvm/guests
>           Product: Virtualization
>           Version: unspecified
>    Kernel Version: 5.1.0-rc6
>          Hardware: Intel
>                OS: Linux
>              Tree: Mainline
>            Status: NEW
>          Severity: blocking
>          Priority: P1
>         Component: kvm
>          Assignee: virtualization_kvm@kernel-bugs.osdl.org
>          Reporter: hilld@binarystorm.net
>        Regression: No
> 
> 1. Please describe the problem:
> Starting with kernel 5.1.0-rc6,  kvm_intel can no longer be loaded in nested
> kvm/guests
> 
> [root@undercloud-0-rhosp10 ~]# modprobe kvm_intel
> modprobe: ERROR: could not insert 'kvm_intel': Input/output error
> 
> 
> 2. What is the Version-Release number of the kernel:
> 5.1.0-rc7
> 
> 3. Did it work previously in Fedora? If so, what kernel version did the issue
>   *first* appear?  Old kernels are available for download at
>  
>   https://urldefense.proofpoint.com/v2/url?u=https-3A__koji.fedoraproject.org_koji_packageinfo-3FpackageID-3D8&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=cB5pEya2zKbSTvVpkeIHKYlQ1F9qW__mnLe0hXnlBCM&s=1wtgL9MEqhN6ZwOZRMKlcW6LYP3zCgz4-1lh8aXyTWo&e=
>   :
> Yes it seems to have appearded between 5.1.0-rc4 (it worked) and 5.1.0-rc6
> (it
> no longer worked)
> 
> 4. Can you reproduce this issue? If so, please provide the steps to reproduce
>   the issue below:
> Yes, update to 5.1.0-rc6 and try to modprobe kvm_intel inside a guest where
> the
> VMX capabilities has been exposted
> 
> 
> 5. Does this problem occur with the latest Rawhide kernel? To install the
>   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
>   ``sudo dnf update --enablerepo=rawhide kernel``:
> Yes
> 
> 
> 6. Are you running any modules that not shipped with directly Fedora's
> kernel?:
> No
> 
> 7. Please attach the kernel logs. You can get the complete kernel log
>   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
>   issue occurred on a previous boot, use the journalctl ``-b`` flag.
> 
> -- 
> You are receiving this mail because:
> You are watching the assignee of the bug.

Comment 3 David Hill 2019-05-08 12:44:05 UTC

The guest vmx CPU flag is exposed like this:

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>IvyBridge-IBRS</model>
    <feature policy='require' name='vmx'/>
  </cpu>


and here the the "cpuid -r" from the guest :

CPU 0:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306a9 ebx=0x00000800 ecx=0xffb82223 edx=0x078bfbff
   0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006
   0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000000
   0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d
   0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800
   0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33
   0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972
   0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000
   0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140
   0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 1:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306a9 ebx=0x01000800 ecx=0xffb82223 edx=0x078bfbff
   0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006
   0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000001
   0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000001
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d
   0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800
   0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33
   0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972
   0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000
   0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140
   0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 2:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306a9 ebx=0x02000800 ecx=0xffb82223 edx=0x078bfbff
   0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006
   0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000002
   0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000002
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d
   0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800
   0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33
   0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972
   0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000
   0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140
   0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 3:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306a9 ebx=0x03000800 ecx=0xffb82223 edx=0x078bfbff
   0x00000002 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x0000004d edx=0x002c307d
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x01: eax=0x00000122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000001
   0x00000004 0x02: eax=0x00000143 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x00000004 0x03: eax=0x00000163 ebx=0x03c0003f ecx=0x00003fff edx=0x00000006
   0x00000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x04000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000001 ecx=0x00000100 edx=0x00000003
   0x0000000b 0x01: eax=0x00000000 ebx=0x00000001 ecx=0x00000201 edx=0x00000003
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x40000000 0x00: eax=0x40000001 ebx=0x4b4d564b ecx=0x564b4d56 edx=0x0000004d
   0x40000001 0x00: eax=0x0100007b ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x80000001 0x00: eax=0x000306a9 ebx=0x00000000 ecx=0x00000001 edx=0x28100800
   0x80000002 0x00: eax=0x65746e49 ebx=0x6558206c ecx=0x45206e6f edx=0x32312d33
   0x80000003 0x00: eax=0x76207878 ebx=0x49282032 ecx=0x42207976 edx=0x67646972
   0x80000004 0x00: eax=0x49202c65 ebx=0x29535242 ecx=0x00000000 edx=0x00000000
   0x80000005 0x00: eax=0x01ff01ff ebx=0x01ff01ff ecx=0x40020140 edx=0x40020140
   0x80000006 0x00: eax=0x00000000 ebx=0x42004200 ecx=0x02008140 edx=0x00808140
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000

Comment 4 David Hill 2019-05-08 12:46:19 UTC

I didn't do anything fancy... only enabled nested kvm with:


[root@wolfe modprobe.d]# cat kvm.conf 
###
### This configuration file was provided by the qemu package.
### Feel free to update as needed.
###

###
### Set these options to enable nested virtualization
###

#options kvm_intel nested=1
#options kvm_amd nested=1
options kvm_intel nested=1

and exposed the flag in the libvirtd kvm definition.   Here is the hardware description of the VM:

  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.0'>hvm</type>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>IvyBridge-IBRS</model>
    <feature policy='require' name='vmx'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>

Comment 5 David Hill 2019-05-08 13:51:40 UTC

I can confirm that reverting that commit solves the problem:

e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”)

Comment 6 Sean Christopherson 2019-05-08 16:10:24 UTC

Revert sent, the previous KVM behavior is correct.  The failing unit test that motivated the change needs to be modified to gracefully handle the #GP or avoid it entirely.

Comment 7 Sean Christopherson 2019-05-08 16:29:07 UTC

On Wed, May 08, 2019 at 07:00:54PM +0300, Liran Alon wrote:
> +Paolo
> 
> What are your thoughts on this?  What is the reason that KVM relies on
> CPU_BASED_RDPMC_EXITING to be exposed from underlying CPU? How is it critical
> for it’s functionality?  If it’s because we want to make sure that we hide
> host PMCs, we should condition this to be a min requirement of kvm_intel only
> in case underlying CPU exposes PMU to begin with.  Do you agree? If yes, I
> can create the patch to fix this.

I sent a revert of the change to hide CPU_BASED_RDPMC_EXITING, KVM's
previous behavior is correct.  The RDPMC instruction was introduced long
before Architctural Perf Mon and so the existence of the exiting control
is dependent only on the instruction, e.g. P4 (Prescott), Core (Yonah)
and Core2 (Merom) all support VMX and RDPMC with non-archictectural
perf mon capabilities.

The KVM unit test first execute RDPMC with interception disabled in the
unit test host, i.e. the #GP is the correct architectural behavior and
needs to be handled by the unit test.  The most robust fix would be to
eat any #GP on RDPMC in the unit test, though it's likely much simpler
to only execute RDPMC with interception disabled if arch perf mon is
supported.

Comment 8 Liran Alon 2019-05-08 22:14:24 UTC

+Paolo

What are your thoughts on this?
What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed from underlying CPU? How is it critical for it’s functionality?
If it’s because we want to make sure that we hide host PMCs, we should condition this to be a min requirement of kvm_intel only in case underlying CPU exposes PMU to begin with.
Do you agree? If yes, I can create the patch to fix this.

-Liran

> On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote:
> 
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e=
> 
> --- Comment #5 from David Hill (hilld@binarystorm.net) ---
> I can confirm that reverting that commit solves the problem:
> 
> e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU”)
> 
> -- 
> You are receiving this mail because:
> You are watching the assignee of the bug.

Comment 9 dhill 2019-05-21 12:57:24 UTC

Hi guys,

    I justed tested kernel 5.2.1-rc1 which contains the following commit 
reverting the previously mentionned commit but... the problem is still 
there:

[root@wolfe linux-stable]# su - jenkins -s /bin/bash
[jenkins@wolfe ~]$ ssh root@192.168.122.2
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Tue May 21 08:53:28 2019 from 192.168.122.1
[root@undercloud-0-rhosp15 ~]# modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error
[root@undercloud-0-rhosp15 ~]# logout
Connection to 192.168.122.2 closed.
[jenkins@wolfe ~]$ uname -a
Linux wolfe.orion 5.2.0-20190521081919+ #6 SMP Tue May 21 08:25:14 EDT 
2019 x86_64 x86_64 x86_64 GNU/Linux

I'll try reverting the commit and then reverting only the commit that 
causes issues.

commit f93f7ede087f2edcc18e4b02310df5749a6b5a61
Author: Sean Christopherson <sean.j.christopherson@intel.com>
Date:   Wed May 8 09:08:19 2019 -0700

     Revert "KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU"

     The RDPMC-exiting control is dependent on the existence of the RDPMC
     instruction itself, i.e. is not tied to the "Architectural Performance
     Monitoring" feature.  For all intents and purposes, the control exists
     on all CPUs with VMX support since RDPMC also exists on all VCPUs with
     VMX supported.  Per Intel's SDM:

       The RDPMC instruction was introduced into the IA-32 Architecture in
       the Pentium Pro processor and the Pentium processor with MMX 
technology.
       The earlier Pentium processors have performance-monitoring 
counters, but
       they must be read with the RDMSR instruction.

     Because RDPMC-exiting always exists, KVM requires the control and 
refuses
     to load if it's not available.  As a result, hiding the PMU from a 
guest
     breaks nested virtualization if the guest attemts to use KVM.

     While it's not explicitly stated in the RDPMC pseudocode, the VM-Exit
     check for RDPMC-exiting follows standard fault vs. VM-Exit 
prioritization
     for privileged instructions, e.g. occurs after the CPL/CR0.PE/CR4.PCE
     checks, but before the counter referenced in ECX is checked for 
validity.

     In other words, the original KVM behavior of injecting a #GP was 
correct,
     and the KVM unit test needs to be adjusted accordingly, e.g. eat 
the #GP
     when the unit test guest (L3 in this case) executes RDPMC without
     RDPMC-exiting set in the unit test host (L2).

     This reverts commit e51bfdb68725dc052d16241ace40ea3140f938aa.

     Fixes: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when 
guest supports PMU")
     Reported-by: David Hill <hilld@binarystorm.net>
     Cc: Saar Amar <saaramar@microsoft.com>
     Cc: Mihai Carabas <mihai.carabas@oracle.com>
     Cc: Jim Mattson <jmattson@google.com>
     Cc: Liran Alon <liran.alon@oracle.com>
     Cc: stable@vger.kernel.org
     Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

On 2019-05-08 6:14 p.m., bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=203543
>
> --- Comment #8 from Liran Alon (liran.alon@oracle.com) ---
> +Paolo
>
> What are your thoughts on this?
> What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed
> from underlying CPU? How is it critical for it’s functionality?
> If it’s because we want to make sure that we hide host PMCs, we should
> condition this to be a min requirement of kvm_intel only in case underlying
> CPU
> exposes PMU to begin with.
> Do you agree? If yes, I can create the patch to fix this.
>
> -Liran
>
>> On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote:
>>
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e=
>>
>> --- Comment #5 from David Hill (hilld@binarystorm.net) ---
>> I can confirm that reverting that commit solves the problem:
>>
>> e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports
>> PMU”)
>>
>> -- 
>> You are receiving this mail because:
>> You are watching the assignee of the bug.

Comment 10 moi 2019-05-21 13:37:42 UTC

Reverting both commits solves this problem:

f93f7ede087f2edcc18e4b02310df5749a6b5a61
e51bfdb68725dc052d16241ace40ea3140f938aa

On 2019-05-21 8:57 a.m., bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=203543
>
> --- Comment #9 from dhill@redhat.com ---
> Hi guys,
>
>      I justed tested kernel 5.2.1-rc1 which contains the following commit
> reverting the previously mentionned commit but... the problem is still
> there:
>
>
> [root@wolfe linux-stable]# su - jenkins -s /bin/bash
> [jenkins@wolfe ~]$ ssh root@192.168.122.2
> Activate the web console with: systemctl enable --now cockpit.socket
>
> Last login: Tue May 21 08:53:28 2019 from 192.168.122.1
> [root@undercloud-0-rhosp15 ~]# modprobe kvm_intel
> modprobe: ERROR: could not insert 'kvm_intel': Input/output error
> [root@undercloud-0-rhosp15 ~]# logout
> Connection to 192.168.122.2 closed.
> [jenkins@wolfe ~]$ uname -a
> Linux wolfe.orion 5.2.0-20190521081919+ #6 SMP Tue May 21 08:25:14 EDT
> 2019 x86_64 x86_64 x86_64 GNU/Linux
>
>
> I'll try reverting the commit and then reverting only the commit that
> causes issues.
>
> commit f93f7ede087f2edcc18e4b02310df5749a6b5a61
> Author: Sean Christopherson <sean.j.christopherson@intel.com>
> Date:   Wed May 8 09:08:19 2019 -0700
>
>       Revert "KVM: nVMX: Expose RDPMC-exiting only when guest supports PMU"
>
>       The RDPMC-exiting control is dependent on the existence of the RDPMC
>       instruction itself, i.e. is not tied to the "Architectural Performance
>       Monitoring" feature.  For all intents and purposes, the control exists
>       on all CPUs with VMX support since RDPMC also exists on all VCPUs with
>       VMX supported.  Per Intel's SDM:
>
>         The RDPMC instruction was introduced into the IA-32 Architecture in
>         the Pentium Pro processor and the Pentium processor with MMX
> technology.
>         The earlier Pentium processors have performance-monitoring
> counters, but
>         they must be read with the RDMSR instruction.
>
>       Because RDPMC-exiting always exists, KVM requires the control and
> refuses
>       to load if it's not available.  As a result, hiding the PMU from a
> guest
>       breaks nested virtualization if the guest attemts to use KVM.
>
>       While it's not explicitly stated in the RDPMC pseudocode, the VM-Exit
>       check for RDPMC-exiting follows standard fault vs. VM-Exit
> prioritization
>       for privileged instructions, e.g. occurs after the CPL/CR0.PE/CR4.PCE
>       checks, but before the counter referenced in ECX is checked for
> validity.
>
>       In other words, the original KVM behavior of injecting a #GP was
> correct,
>       and the KVM unit test needs to be adjusted accordingly, e.g. eat
> the #GP
>       when the unit test guest (L3 in this case) executes RDPMC without
>       RDPMC-exiting set in the unit test host (L2).
>
>       This reverts commit e51bfdb68725dc052d16241ace40ea3140f938aa.
>
>       Fixes: e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when
> guest supports PMU")
>       Reported-by: David Hill <hilld@binarystorm.net>
>       Cc: Saar Amar <saaramar@microsoft.com>
>       Cc: Mihai Carabas <mihai.carabas@oracle.com>
>       Cc: Jim Mattson <jmattson@google.com>
>       Cc: Liran Alon <liran.alon@oracle.com>
>       Cc: stable@vger.kernel.org
>       Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>       Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
>
> On 2019-05-08 6:14 p.m., bugzilla-daemon@bugzilla.kernel.org wrote:
>> https://bugzilla.kernel.org/show_bug.cgi?id=203543
>>
>> --- Comment #8 from Liran Alon (liran.alon@oracle.com) ---
>> +Paolo
>>
>> What are your thoughts on this?
>> What is the reason that KVM relies on CPU_BASED_RDPMC_EXITING to be exposed
>> from underlying CPU? How is it critical for it’s functionality?
>> If it’s because we want to make sure that we hide host PMCs, we should
>> condition this to be a min requirement of kvm_intel only in case underlying
>> CPU
>> exposes PMU to begin with.
>> Do you agree? If yes, I can create the patch to fix this.
>>
>> -Liran
>>
>>> On 8 May 2019, at 16:51, bugzilla-daemon@bugzilla.kernel.org wrote:
>>>
>>>
>>>
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.kernel.org_show-5Fbug.cgi-3Fid-3D203543&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=7TirfLMNxYI-3Ygxm3kjDUB49Jwmk8bqD7671wy0hi8&s=Z_L1UqH19zon0ohDrCMU91ixA-Wn_vO7d-fO8s2G3PI&e=
>>>
>>> --- Comment #5 from David Hill (hilld@binarystorm.net) ---
>>> I can confirm that reverting that commit solves the problem:
>>>
>>> e51bfdb68725 ("KVM: nVMX: Expose RDPMC-exiting only when guest supports
>>> PMU”)
>>>
>>> -- 
>>> You are receiving this mail because:
>>> You are watching the assignee of the bug.

Comment 11 Sean Christopherson 2019-05-21 14:17:33 UTC

On Tue, May 21, 2019 at 07:11:01AM -0700, Sean Christopherson wrote:
> On Tue, May 21, 2019 at 01:37:42PM +0000, bugzilla-daemon@bugzilla.kernel.org
> wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=203543
> > 
> > --- Comment #10 from moi@davidchill.ca ---
> > Reverting both commits solves this problem:
> > 
> > f93f7ede087f2edcc18e4b02310df5749a6b5a61
> > e51bfdb68725dc052d16241ace40ea3140f938aa
> 
> Hmm, that makes no sense, f93f7ede087f is a straight revert of
> e51bfdb68725.  I do see the same behavior on v5.2-rc1 where hiding the
> pmu from L1 breaks nested virtualization, but manually reverting both
> commits doesn't change that for me, i.e. there's another bug lurking,
> which I'll start hunting.

Scratch that, had a brain fart and tested the wrong kernel.  I do *NOT*
see breakage on v5.2-rc1, at least when running v5.2-rc1 as L1 and probing
KVM in L2.

When running v5.2-rc1 as L0, what are the values of MSRs 0x482 and 0x48e
in L1?

> 
> Any chance the successful test used a different command line or something?

Comment 12 Sean Christopherson 2019-05-21 14:18:10 UTC

On Tue, May 21, 2019 at 01:37:42PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=203543
> 
> --- Comment #10 from moi@davidchill.ca ---
> Reverting both commits solves this problem:
> 
> f93f7ede087f2edcc18e4b02310df5749a6b5a61
> e51bfdb68725dc052d16241ace40ea3140f938aa

Hmm, that makes no sense, f93f7ede087f is a straight revert of
e51bfdb68725.  I do see the same behavior on v5.2-rc1 where hiding the
pmu from L1 breaks nested virtualization, but manually reverting both
commits doesn't change that for me, i.e. there's another bug lurking,
which I'll start hunting.

Any chance the successful test used a different command line or something?

Comment 13 David Hill 2019-05-21 16:02:56 UTC

Well, I did this:

1) git checkout v5.2-rc1
2) compile it
3) configure grub to use it
4) reboot
5) boot the existing VM
6) lsmod | grep kvm_intel

and at 5) it wasn't loaded.  next I did:
1) git checkout v5.2-rc1
2) git revert f93f7ede087f2edcc18e4b02310df5749a6b5a61
3) git revert e51bfdb68725dc052d16241ace40ea3140f938aa.
4) compile that
5) configure grub to use it
6) reboot
7) boot the existing VM
8) lsmod | grep kvm_intel 

and at 8) it was loaded.

Comment 14 Sean Christopherson 2019-05-21 18:06:18 UTC

I've verified reverting f93f7ede087f and e51bfdb68725 yields the exact same code as v5.2-rc1.  Can you please sanity check that v5.2-rc1 is indeed broken?  E.g. ensure there are no modified files when compiling and include the git commit id in the kernel name.

Comment 15 dhill 2019-05-26 12:11:34 UTC

You are right actually.   I did a mistake in my code and it didn't pull 
v5.2-rc1 ... I was still on v5.1.x tag so that's why it didn't work.

I guess this is issue is solved and if I hit any bugs with v5.2-rc1 it'd 
be a new issue since we reverted that commit.

On 2019-05-21 2:06 p.m., bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=203543
>
> --- Comment #14 from Sean Christopherson (sean.j.christopherson@intel.com)
> ---
> I've verified reverting f93f7ede087f and e51bfdb68725 yields the exact same
> code as v5.2-rc1.  Can you please sanity check that v5.2-rc1 is indeed
> broken?
> E.g. ensure there are no modified files when compiling and include the git
> commit id in the kernel name.
>

Comment 16 David Hill 2019-12-01 18:07:19 UTC

Looks like this issue is back :

[root@wolfe linux-stable]# uname -a
Linux wolfe.orion 5.4.0-2.fc32.x86_64 #1 SMP Mon Nov 25 22:45:19 UTC 2019 x86_64 


[root@undercloud-0-rhosp13 ~]# modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error

Comment 17 David Hill 2019-12-01 21:49:36 UTC

Should I open a new bug for this issue ?

Comment 18 David Hill 2019-12-02 03:20:39 UTC

v5.3.2 appears to be working

Comment 19 David Hill 2019-12-02 03:42:50 UTC

v5.3.14 appears to be working too ... if I login another VM, I do get the same error message:

[root@overcloud-compute-0 ~]# modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error



(undercloud) [stack@undercloud-0-rhosp13 ~]$ sudo lsmod | grep kvm
kvm_intel             188688  0 
kvm                   636931  1 kvm_intel
irqbypass              13503  1 kvm

Comment 20 David Hill 2019-12-02 04:15:24 UTC

I got an intermittent 

[root@undercloud-0-rhosp13 ~]# modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error

even with 5.1.0-rc4.  This is getting stranger.

Comment 21 David Hill 2019-12-02 04:27:34 UTC

Something is constant here and it's that if I was able to load kvm_intel on one of the VMs, no other VMs can modprobe kvm_intel.

Comment 22 David Hill 2019-12-02 04:44:21 UTC

Ok with latest kernel and libvirt-5.9.0-1.fc32.x86_64, nested virt is broken but if I downgrade to libvirt-5.6.0-4.fc31.x86_64, problem is solved.  It looks like nested virt is broken in libvirt-5.9.0-1.

Sorry for the noise.  I'll close this bug.