Bug 42636

           +-14.4-[05]--+-06.0  Intel Corporation 82557/8/9/0/1 Ethernet Pro100
           |            \-07.0  RaLink RT2800 802.11n PCI

[    0.610382] AMD-Vi:   DEV_SELECT                      devid: 00:14.4 flags:00
[    0.610384] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 05:00.0 flags:00 devid_to: 00:14.4
[    0.610385] AMD-Vi:   DEV_RANGE_END           devid: 05:1f.7

The devices are behind a PCIe-to-PCI bridge (00:14.4), so both devices get aliased to the same devices.  You'll need to either add both devices to the guest or sequester the other device by binding it to pci-stub.

Well, I did exactly what you proposed, but I got the same error again, as I tried to apply both devices. That's the relevant part of the xml file:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x06' function='0x0'/>
      </source>
    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x07' function='0x0'/>
      </source>
    </hostdev>

Did I make a mistake?

Hi,
I feel with you - I had many, many headaches with PCI-PTrough...

You said you unloaded the driver module for both devices. 
a) do you have the pcistub module available (kernel config)? For some PCI devices I had to enable both DMA and IRQ remapping in the kernel. The latter is disabled by default. Otherwise: sometimes, especially with older PCI cards, IRQ remapping had to be disabled... at least as module or compiled-in:

Bus options (PCI etc.) --> Support for DMA Remapping Devices
Bus options (PCI etc.) --> Enable DMA Remapping Devices
Bus options (PCI etc.) --> PCI Stub driver

[quote]
Of course, I unloaded the module of this additional device
before trying to passthrough it to the guest.[/qoute]

b) Did you try to assign the pcistub driver as printed here:
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Created attachment 72753 [details]
log of qemu-kvm crash

I tried again with only one pci card inserted (Network controller: RaLink RT2800 802.11n PCI). The tree is now like this:

           +-14.4-[06]----07.0  RaLink RT2800 802.11n PCI

I booted with intel_iommu=on (does this apply to AMD, too?).

If I start the VM now, I'm getting the same error again:
Failed to assign device "hostdev0" : Device or resource busy
qemu-kvm: -device pci-assign,host=06:07.0,id=hostdev0,configfd=19,bus=pci.0,addr=0x5: Device 'pci-assign' could not be initialized

If I'm additionally putting the bridge to the VM, I'm getting an error telling me about a missing reset function and another missing thing, too.

If I remove the bridge from the VM xml-file again and start it now, the VM suddenly starts up fine :-))).

But if the VM is stopped again (with halt in the VM or with virsh shutdown VM, qemu-kvm crashes and remains as zombie. The output from messages is attached as iommu-trace.log.

kvm is 0.15.0 and kernel is 3.1.10 (kernel-desktop from openSUSE)

More specific error message while VM is started with attached PCI bridge:

Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset available

I tried with kvm module for linux 3.3 and got the same crash while stopping the VM.

Hi, 
for amd, pass

"iommu=pt iommu=1" to kernel options

you can also try to load kvm module with allow_unsafe_assigned_interrupts=1

echo "options kvm allow_unsafe_assigned_interrupts=1" > /etc/modprobe.d/kvm_iommu_map_guest.conf

Thanks,
with the "iommu=pt iommu=1" options applied it doesn't work at all - even the "workaround" with first try to start with the bridge applied and afterwards without the bridge added isn't working then.

BTW: I can pass through PCIe devices, as long as they have different PCI ids and different modules, without any problem.

I'm using the option allow_unsafe_assigned_interrupts=1 already!

I think there is a problem with the handling of the device behind a PCI bridge, which should be fixed (-> the Ralink device which should be passed through is a PCI device and not a PCIe device)! Take a look at the qemu-kvm crash log https://bugzilla.kernel.org/attachment.cgi?id=72753

I have same problems with pci passthrough, while my pci-e video card hd7750 with hda(01:00.0/1) and integrated network card(05:00.0) succsessfilly assigned in guest, when i try to add usb controller, i get same error about FLR:

Unable to reset PCI device 0000:00:12.0: no FLR, PM reset or bus reset
available

with xen it works fine.

lspci -tv
-[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B)
           +-00.2  Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU)
           +-02.0-[01]--+-00.0  Advanced Micro Devices [AMD] nee ATI Device 683f
           |            \-00.1  Advanced Micro Devices [AMD] nee ATI Device aab0
           +-09.0-[02]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host Controller
           +-0b.0-[03]--+-00.0  nVidia Corporation GF110 [GeForce GTX 560 Ti]
           |            \-00.1  nVidia Corporation GF110 High Definition Audio Controller

           +-11.0  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]
           +-12.0  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
           +-12.2  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
           +-13.0  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
           +-13.2  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
           +-14.0  Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller
           +-14.2  Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA)
           +-14.3  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller
           +-14.4-[04]--
           +-14.5  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
           +-15.0-[05]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
           +-15.1-[06]--
           +-15.2-[07]--
           +-15.3-[08]--
           +-16.0  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
           +-16.2  Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
           +-18.0  Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration
           +-18.1  Advanced Micro Devices [AMD] Family 10h Processor Address Map
           +-18.2  Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller
           +-18.3  Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control
           \-18.4  Advanced Micro Devices [AMD] Family 10h Processor Link Control


qemu-kvm-1.0, linux-3.3
motherboard is GA-990FXA-D3

I tested the same device here and got the same error as you reported.

Good to know, that Xen doesn't have any problem. This really means, that it is most probably a kvm or qemu or AMD iommu bug.

kvm 0.15, kernel: linux 3.1.10 (openSUSE)