Bug 208091

Summary:	vcpu1, guest rIP offset ignored wrmsr or rdmsr
Product:	Virtualization	Reporter:	Joris L. (commandline)
Component:	kvm	Assignee:	virtualization_kvm
Status:	RESOLVED DOCUMENTED
Severity:	normal	CC:	seanjc
Priority:	P1
Hardware:	x86-64
OS:	Linux
URL:	https://patchwork.kernel.org/patch/7670771/
Kernel Version:	Linux pvx 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) x86_64 GNU/Linux	Subsystem:
Regression:	No	Bisected commit-id:

Description Joris L. 2020-06-06 17:09:26 UTC

On launch of VM these messages are generated. Filing this report since hinted at a reason for improvement such as for patch 7670771. First ever report, sorry if violating any good habits.



pvx kernel: [   77.102686] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x3a
pvx kernel: [   77.102712] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0xd90
pvx kernel: [   77.102736] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x570
pvx kernel: [   77.102755] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x571
pvx kernel: [   77.102773] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x572
pvx kernel: [   77.102792] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x560
pvx kernel: [   77.102811] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x561
pvx kernel: [   77.102829] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x580
pvx kernel: [   77.102847] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x581
pvx kernel: [   77.102866] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x582

pvx kernel: [   77.102686] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x3a
pvx kernel: [   77.102712] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0xd90
pvx kernel: [   77.102736] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x570
pvx kernel: [   77.102755] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x571
pvx kernel: [   77.102773] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x572
pvx kernel: [   77.102792] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x560
pvx kernel: [   77.102811] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x561
pvx kernel: [   77.102829] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x580
pvx kernel: [   77.102847] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x581
pvx kernel: [   77.102866] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4 ignored rdmsr: 0x582

Comment 1 Joris L. 2020-06-06 17:37:07 UTC

iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99eeb0]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef10]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef50]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef70]
vfio-pci 0000:28:00.0: timed out waiting for pending transaction; performing function level reset anyway

device=DD < anonimised

Comment 2 Joris L. 2020-06-08 15:18:32 UTC

More of the same

[  507.188509] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[  507.188524] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[  513.661635] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0aa ignored rdmsr: 0xc0011023
[  513.661644] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0bf ignored wrmsr: 0xc0011023 data 0x100
[  513.703575] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0aa ignored rdmsr: 0xc0011023
[  513.703588] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0bf ignored wrmsr: 0xc0011023 data 0x100
[  513.704769] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0aa ignored rdmsr: 0xc0011023
[  513.704782] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0bf ignored wrmsr: 0xc0011023 data 0x100
[  513.705948] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0aa ignored rdmsr: 0xc0011023
[  513.705961] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0bf ignored wrmsr: 0xc0011023 data 0x100

Comment 3 Sean Christopherson 2020-06-08 15:50:31 UTC

The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is not the default.  Is that enabled for any particular reason?  Or maybe a better question is, what is it that you're trying to do and what, if any, errors are occurring?

E.g. in the first trace, it looks like you're advertising an Intel vCPU to the guest on top of AMD hardware.  That can be made to work, but it's not recommended.  Ignoring MSRs in that scenario is almost guaranteed to cause weirdness.

Comment 4 Joris L. 2020-06-08 15:55:00 UTC

These message are observed when starting a Proxmox VE 6.2 VM guest with GPU passthrough.  Current configuration is at the point there are not other error messages visible related to VM activity.

Available to respond to and provide for actions to improve documenting these messages etc.

The vcpu0, guest rIP messages persist while a vm is running. Stopping and staring a VM results in no screen. messages however now show (repeat kvm ... msr messages, repeeat block of kvm, vcpu... rIP messages) followed by ecap messages.

[ 2762.744734] kvm_get_msr_common: 275 callbacks suppressed
[ 2762.744738] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010064
[ 2762.744775] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010065
[ 2762.744853] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010066
[ 2762.744885] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010067
[ 2762.744917] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010068
[ 2762.744948] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010069
[ 2762.744983] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc001006a
[ 2762.745014] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc001006b
[ 2762.745044] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xc0010293
[ 2762.750132] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr: 0xe8
[ 2844.034627] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[ 2844.034641] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[ 2845.308346] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.324311] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.566461] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.569777] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.599693] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.602566] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.625243] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.627436] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.659282] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.665352] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.673673] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.673924] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.673948] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.674104] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.674870] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.675040] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.675063] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.675210] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.708785] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.710674] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.727404] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.727598] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.728296] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.728484] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.764636] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.765256] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2845.933638] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery - restoring BARs

now the VM no longer displays the desktop.

Comment 5 Joris L. 2020-06-08 16:06:08 UTC

(In reply to Sean Christopherson from comment #3)
> The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is
> not the default.  Is that enabled for any particular reason?  Or maybe a
> better question is, what is it that you're trying to do and what, if any,
> errors are occurring?
> 
> E.g. in the first trace, it looks like you're advertising an Intel vCPU to
> the guest on top of AMD hardware.  That can be made to work, but it's not
> recommended.  Ignoring MSRs in that scenario is almost guaranteed to cause
> weirdness.

Thanks Sean. I am cleaning up this config. Will check.

This is indeed a Ryzen CPU with Qemu/Proxmox on top of it.
For GPU Passthrough to a VM the kvm.ignore_msrs=1 is often recommended.
I brief test shows the VM is not working properly without this parameter, even when booting the VM with CPU:host.

br,

Joris

Comment 6 Joris L. 2020-06-08 16:15:03 UTC

These i get with kvm ignore.msrs=1 enabled and CPU=host

[   46.512153] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0aa ignored rdmsr: 0xc0011023
[   46.512161] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0bf ignored wrmsr: 0xc0011023 data 0x100
[   46.555401] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0aa ignored rdmsr: 0xc0011023
[   46.555406] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0bf ignored wrmsr: 0xc0011023 data 0x100
[   46.556809] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0aa ignored rdmsr: 0xc0011023
[   46.556823] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0bf ignored wrmsr: 0xc0011023 data 0x100
[   46.558065] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0aa ignored rdmsr: 0xc0011023
[   46.558078] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0bf ignored wrmsr: 0xc0011023 data 0x100

Comment 7 Joris L. 2020-06-08 19:15:19 UTC

With the kvm ignor.msrs parameter removed i do indeed see different results. Now i am back at these messages. Sorry for the ignorance, learning as i read the feedback.

[10427.220982] vfio-pci 0000:17:00.0: timed out waiting for pending transaction; performing function level reset anyway
[10428.469006] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10429.524940] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10431.668982] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10436.020672] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting
[10444.469003] vfio-pci 0000:17:00.0: not ready 16383ms after FLR; waiting
[10462.133026] vfio-pci 0000:17:00.0: not ready 32767ms after FLR; waiting
[10496.948661] vfio-pci 0000:17:00.0: not ready 65535ms after FLR; giving up

Comment 8 Joris L. 2020-06-08 19:23:02 UTC

followed by


[10956.384880] vfio-pci 0000:17:00.0: Refused to change power state, currently in D3
[10957.109026] vfio-pci 0000:17:00.0: timed out waiting for pending transaction; performing function level reset anyway
[10958.356694] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10959.412727] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10961.588924] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10965.940829] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting

Comment 9 Joris L. 2020-06-19 22:24:00 UTC

eventually, boot time parameters resolved all of these issues, sharing in case they matter.

iommu=pt amd_iommu=forced_isolation video=vesafb:off,efifb:off rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction pcie_pme=nomsi pcie_aspm=force pcie_ports=native