Created attachment 145421 [details]
crash_netconsole.txt

Created attachment 145431 [details]
startup_dmesg.txt

Created attachment 145441 [details]
dmidecode.txt

Also occurs with freshly built mainline kernel version 3.16.0.

[   87.327457] ------------[ cut here ]------------
[   87.327488] kernel BUG at drivers/iommu/amd_iommu.c:2382!
[   87.327505] invalid opcode: 0000 [#1] SMP 
[   87.327526] Modules linked in: pci_stub(E) ipt_MASQUERADE(E) iptable_nat(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) xt_conntrack(E) nf_conntrack(E) ipt_REJECT(E) xt_CHECKSUM(E) iptable_mangle(E) xt_tcpudp(E) bridge(E) stp(E) llc(E) ip6table_filter(E) ip6_tables(E) iptable_filter(E) ip_tables(E) ebtable_nat(E) ebtables(E) x_tables(E) nct6775(E) hwmon_vid(E) radeon(E) kvm_amd(E) kvm(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_codec_hdmi(E) snd_hda_intel(E) snd_hda_controller(E) snd_hda_codec(E) i2c_algo_bit(E) crct10dif_pclmul(E) drm_kms_helper(E) crc32_pclmul(E) ghash_clmulni_intel(E) snd_hwdep(E) aesni_intel(E) snd_pcm(E) ttm(E) aes_x86_64(E) glue_helper(E) netconsole(E) drm(E) lrw(E) snd_timer(E) configfs(E) snd(E) gf128mul(E) ablk_helper(E) cryptd(E) soundcore(E) lp(E) serio_raw(E) k10temp(E) i2c_piix4(E) mac_hid(E) video(E) parport(E) usb_storage(E) pata_acpi(E) hid_generic(E) usbhid(E) hid(E) alx(E) psmouse(E) mdio(E) pata_atiixp(E) ahci(E) libahci(E)
[   87.327963] CPU: 0 PID: 1452 Comm: qemu-system-x86 Tainted: G            E 3.16.0 #1
[   87.327986] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A88X Extreme6+, BIOS L3.16 04/16/2014
[   87.328016] task: ffff880427a18000 ti: ffff880421280000 task.ti: ffff880421280000
[   87.328039] RIP: 0010:[<ffffffff816059dd>]  [<ffffffff816059dd>] __detach_device+0xad/0xb0
[   87.328071] RSP: 0018:ffff880421283b38  EFLAGS: 00010046
[   87.328088] RAX: 0000000000000000 RBX: ffff8804286e5240 RCX: ffff880421283ae0
[   87.328110] RDX: dead000000100100 RSI: 0000000000000086 RDI: ffff8804286e5240
[   87.328132] RBP: ffff880421283b58 R08: 0000000000000046 R09: ffff8804299b8900
[   87.328154] R10: ffff880000000000 R11: 000ffffffffff000 R12: 0000000000000000
[   87.328175] R13: ffff88042127a610 R14: ffff88042744c040 R15: ffff8804286e5240
[   87.328197] FS:  00007f1d03857700(0000) GS:ffff88043ec00000(0000) knlGS:0000000000000000
[   87.328221] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   87.328239] CR2: 00007f1d03dc63a0 CR3: 0000000001c13000 CR4: 00000000000407f0
[   87.328260] Stack:
[   87.328268]  dead000000100100 ffff88042127a600 ffff88042127a610 ffff88042744c040
[   87.328299]  ffff880421283b98 ffffffff81605a7e 0000000000000202 ffff88042744c040
[   87.328333]  ffff88042744c040 ffff880420a3c008 ffff8804242b0a80 ffff88007786dfd8
[   87.328365] Call Trace:
[   87.328378]  [<ffffffff81605a7e>] amd_iommu_domain_destroy+0x9e/0x160
[   87.328400]  [<ffffffff816022db>] iommu_domain_free+0x1b/0x30
[   87.328432]  [<ffffffffa03628a3>] kvm_iommu_unmap_guest+0x53/0x60 [kvm]
[   87.328461]  [<ffffffffa0373059>] kvm_arch_destroy_vm+0x39/0x1f0 [kvm]
[   87.328484]  [<ffffffff810cfebd>] ? synchronize_srcu+0x1d/0x20
[   87.328509]  [<ffffffffa035b26e>] kvm_put_kvm+0x10e/0x220 [kvm]
[   87.328535]  [<ffffffffa035b3b8>] kvm_vcpu_release+0x18/0x20 [kvm]
[   87.328556]  [<ffffffff811d0a04>] __fput+0xe4/0x220
[   87.328573]  [<ffffffff811d0b8e>] ____fput+0xe/0x10
[   87.328591]  [<ffffffff8108cd74>] task_work_run+0xc4/0xe0
[   87.328609]  [<ffffffff8106ef18>] do_exit+0x2b8/0xa60
[   87.328627]  [<ffffffff8106f73f>] do_group_exit+0x3f/0xa0
[   87.328645]  [<ffffffff8107f100>] get_signal_to_deliver+0x1d0/0x6f0
[   87.328668]  [<ffffffff81012548>] do_signal+0x48/0x9d0
[   87.328687]  [<ffffffff8111d1bc>] ? acct_account_cputime+0x1c/0x20
[   87.328708]  [<ffffffff810a372b>] ? account_user_time+0x8b/0xa0
[   87.329791]  [<ffffffff810a3cf4>] ? vtime_account_user+0x54/0x60
[   87.330869]  [<ffffffff81012f39>] do_notify_resume+0x69/0xb0
[   87.331950]  [<ffffffff8172b32a>] int_signal+0x12/0x17
[   87.333016] Code: fe ff ff eb b8 66 0f 1f 84 00 00 00 00 00 48 8b 35 69 b0 9a 00 49 39 f4 74 c1 48 89 df e8 8c fd ff ff 5b 41 5c 41 5d 41 5e 5d c3 <0f> 0b 90 66 66 66 66 90 55 48 89 e5 41 57 41 56 49 89 fe 41 55 
[   87.335373] RIP  [<ffffffff816059dd>] __detach_device+0xad/0xb0
[   87.336475]  RSP <ffff880421283b38>
[   87.337562] ---[ end trace bee5733468f37c81 ]---

What if you use vfio-pci instead of pci-assign?  The BUG happens when the kernel tries to detach a device from the domain, but the device doesn't actually belong to a domain.  VFIO likely already avoids this because the bridge and device will both be in the same IOMMU group and therefore attached to the same domain.

(In reply to Alex Williamson from comment #5)
> What if you use vfio-pci instead of pci-assign?

I run into the dreaded error:
  vfio: error, group 9 is not viable, please ensure all devices within the
  iommu_group are bound to their vfio bus driver

There are some proposed workarounds on the web, like passing vfio_iommu_type1.allow_unsafe_interrupts=1 or pci=realloc, but these seem to change nothing for me.

So I tried adding all the PCI devices in the IOMMU group as passthrough devices (including IDE, SMBus, audio and OHCI controllers). But then QEMU's SeaBIOS gets so confused it can no longer find a hard drive to boot off.

But you're right. At least I can stop the non-functional virtual machine now, so I've got that going for me, which is nice.

(In reply to Marti Raudsepp from comment #6)
> (In reply to Alex Williamson from comment #5)
> > What if you use vfio-pci instead of pci-assign?
> 
> I run into the dreaded error:
>   vfio: error, group 9 is not viable, please ensure all devices within the
>   iommu_group are bound to their vfio bus driver
> 
> There are some proposed workarounds on the web, like passing
> vfio_iommu_type1.allow_unsafe_interrupts=1 or pci=realloc, but these seem to
> change nothing for me.

None of these remotely address the issue.  If you're running at least 3.12 there are quirks for the following AMD southbridge components:

 * 1002:4385 SBx00 SMBus Controller
 * 1002:439c SB7x0/SB8x0/SB9x0 IDE Controller
 * 1002:4383 SBx00 Azalia (Intel HDA)
 * 1002:439d SB7x0/SB8x0/SB9x0 LPC host controller
 * 1002:4384 SBx00 PCI to PCI Bridge
 * 1002:4399 SB7x0/SB8x0/SB9x0 USB OHCI2 Controller

If your bridge does not match these, then AMD will need to confirm whether isolation is provided between your devices.  There is an ACS override patch floating around which allows assuming device isolation, but this is generally a bad idea, can introduce obscure bugs, and will not be merged upstream.

> So I tried adding all the PCI devices in the IOMMU group as passthrough
> devices (including IDE, SMBus, audio and OHCI controllers). But then QEMU's
> SeaBIOS gets so confused it can no longer find a hard drive to boot off.

Note that it's not required to assign all the devices, they simply need to be detached from host drivers (ie. bound to pci-stub or vfio-pci).

(In reply to Alex Williamson from comment #7)
> > There are some proposed workarounds on the web
> None of these remotely address the issue.

I see. This page claims so: http://www.ovirt.org/Features/hostdev_passthrough

> there are quirks for the following AMD southbridge components

Nope, mine are 1022:780b, 1022:780c, 1022:780d, 1022:780e, 1022:780f, 1022:7809

> If your bridge does not match these, then AMD will need to confirm whether
> isolation is provided between your devices.

How would I go about confirming that? What are the chances that they care, and provide accurate information to a random person?

> There is an ACS override patch

I already ran across it... https://bugzilla.redhat.com/show_bug.cgi?id=1113399
Would I be any worse off using this, compared to the old kvm pci-assign method?

> Note that it's not required to assign all the devices, they simply need to
> be detached from host drivers (ie. bound to pci-stub or vfio-pci).

Thanks, I will give it a shot tomorrow.

(In reply to Marti Raudsepp from comment #8)
> (In reply to Alex Williamson from comment #7)
> > > There are some proposed workarounds on the web
> > None of these remotely address the issue.
> 
> I see. This page claims so: http://www.ovirt.org/Features/hostdev_passthrough

Sorry, it's wrong.

> > there are quirks for the following AMD southbridge components
> 
> Nope, mine are 1022:780b, 1022:780c, 1022:780d, 1022:780e, 1022:780f,
> 1022:7809
> 
> > If your bridge does not match these, then AMD will need to confirm whether
> > isolation is provided between your devices.
> 
> How would I go about confirming that? What are the chances that they care,
> and provide accurate information to a random person?

AMD would need to confirm it.  IOMMU groups are based on hardware advertised isolation via the PCIe ACS capability.  Without this, or a device specific quirk to take its place, IOMMU groups must assume that peer-to-peer between functions of a multi-function device is possible and therefore that the devices are not isolated.  Chances are that this new chipset in your system is taking the exact same ASICs that were deemed not to do peer-to-peer on previous chipsets, but we need that confirmation from AMD.  Alex Deucher (see MAINTAINERS) may have contacts available that can make that statement.

> > There is an ACS override patch
> 
> I already ran across it...
> https://bugzilla.redhat.com/show_bug.cgi?id=1113399
> Would I be any worse off using this, compared to the old kvm pci-assign
> method?

I think the path forward is to get confirmation from AMD that these function are isolated from each other and add quirks to the kernel.  Then you won't have the device dependencies in vfio-pci.  The override patch allows you to do that with just a kernel boot parameter.  There's no gurantee that pci-assign will ever be fixed since it's being phased out.

(In reply to Alex Williamson from comment #9)
> (In reply to Marti Raudsepp from comment #8)
> > (In reply to Alex Williamson from comment #7)
> > > > There are some proposed workarounds on the web
> > > None of these remotely address the issue.
> > 
> > I see. This page claims so:
> http://www.ovirt.org/Features/hostdev_passthrough
> 
> Sorry, it's wrong.
> 
> > > there are quirks for the following AMD southbridge components
> > 
> > Nope, mine are 1022:780b, 1022:780c, 1022:780d, 1022:780e, 1022:780f,
> > 1022:7809
> > 
> > > If your bridge does not match these, then AMD will need to confirm
> whether
> > > isolation is provided between your devices.
> > 
> > How would I go about confirming that? What are the chances that they care,
> > and provide accurate information to a random person?


Are you suggesting we'd provide innacurate information to a random person?


> 
> AMD would need to confirm it.  IOMMU groups are based on hardware advertised
> isolation via the PCIe ACS capability.  Without this, or a device specific
> quirk to take its place, IOMMU groups must assume that peer-to-peer between
> functions of a multi-function device is possible and therefore that the
> devices are not isolated.  Chances are that this new chipset in your system
> is taking the exact same ASICs that were deemed not to do peer-to-peer on
> previous chipsets, but we need that confirmation from AMD.  Alex Deucher
> (see MAINTAINERS) may have contacts available that can make that statement.

I don't have an answer for you offhand.  Let me do some digging and get you an answer.

> 
> > > There is an ACS override patch
> > 
> > I already ran across it...
> > https://bugzilla.redhat.com/show_bug.cgi?id=1113399
> > Would I be any worse off using this, compared to the old kvm pci-assign
> > method?
> 
> I think the path forward is to get confirmation from AMD that these function
> are isolated from each other and add quirks to the kernel.  Then you won't
> have the device dependencies in vfio-pci.  The override patch allows you to
> do that with just a kernel boot parameter.  There's no gurantee that
> pci-assign will ever be fixed since it's being phased out.

(In reply to Joel Schopp from comment #10)
> > How would I go about confirming that? What are the chances that they care,
> > and provide accurate information to a random person?
> 
> Are you suggesting we'd provide innacurate information to a random person?

Yes, that's my experience with the "customer support" for desktop hardware.

Of course cutting support out of the equation and asking engineers directly is likely to give better results, that didn't occur to me at first.

Created attachment 146311 [details]
Possible fix as a patch against v3.13

Hi Marti,

Can you please test this patch? I think it should fix the issue.

Thanks, Joerg

(In reply to Joerg Roedel from comment #12)
> Thanks, Joerg

Indeed. Thanks, Joerg. And thanks everyone else too, you have been very helpful!

I didn't have v3.13 sources handy, but I applied the attachment 146311 [details] patch to 3.16.0 and it fixes the problem. (I verified that unpatched 3.16.0 also crashes).

I can start & shut down the VM multiple times without crashing the host and PCI passthrough works as expected.

Feel free to add
    Tested-by: Marti Raudsepp <marti@juffo.org>

(In reply to Alex Williamson from comment #7)
> Note that it's not required to assign all the devices, they simply need to
> be detached from host drivers (ie. bound to pci-stub or vfio-pci).

This approach also works; I think I will go this route for the production setup. Seems that we don't actually need any of the devices in the same IOMMU group.

(In reply to Joel Schopp from comment #10)
> > AMD would need to confirm it.
>
> I don't have an answer for you offhand.  Let me do some digging and get you
> an answer.

I am sorry if I sounded frustrated or arrogant earlier. Any update on this?

Hi Marti,

> Indeed. Thanks, Joerg. And thanks everyone else too, you have been very
> helpful!
> 
> I didn't have v3.13 sources handy, but I applied the attachment 146311 [details]

> I can start & shut down the VM multiple times without crashing the host and
> PCI passthrough works as expected.
> 
> Feel free to add
>     Tested-by: Marti Raudsepp <marti@juffo.org>

Thanks for testing the fix, I will send it upstream once the merge window is over -rc1 is released. I also added a stable tag so it gets backported.

Joerg

> (In reply to Joel Schopp from comment #10)
> > > AMD would need to confirm it.
> >
> > I don't have an answer for you offhand.  Let me do some digging and get you
> > an answer.
> 
> I am sorry if I sounded frustrated or arrogant earlier. Any update on this?

It's not clear to me which devices were being put in the same group.  Here's some of my notes on your lspci output

lspci -vt
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1422
           +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1423
           +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Kaveri [Radeon R7 200 Series]
           +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Device 1308
           +-02.0  Advanced Micro Devices, Inc. [AMD] Device 1424
           +-03.0  Advanced Micro Devices, Inc. [AMD] Device 1424
           +-04.0  Advanced Micro Devices, Inc. [AMD] Device 1424
           +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
           +-10.1  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller

These xhci controllers are isolated from from the other devices, I would need some more detail on which variant you are running to determine if they are isolated from eachother, they probably aren't.

           +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
The sata controller is isolated from the other devices

           +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
           +-12.2  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
This pair of OHCI/EHCI controllers are together isolated from the other devices

           +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
           +-13.2  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
This pair of OHCI/EHCI controllers are together isolated from the other devices

           +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
           +-14.1  Advanced Micro Devices, Inc. [AMD] FCH IDE Controller
           +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
           +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
I do not think the SMBus/IDE/Azalia/LPC are isolated from eachother, but they are isolated from the other devices I have identified.


           +-14.4-[01]----05.0  Dialogic Corporation PRI
The legacy PCI should be isolated from the other devices identified.  Not sure what is going on here.

           +-14.5  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
This OHCI Controller should also be isolated from the other devices.

           +-15.0-[02]--
           +-15.2-[03]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
Is this in a PCI-e slot or otherwise attached to the PCI-e?

           +-15.3-[04]----00.0  Qualcomm Atheros QCA8171 Gigabit Ethernet
Is this in a PCI-e slot or otherwise attached to the PCI-e?

           +-18.0  Advanced Micro Devices, Inc. [AMD] Device 141a
           +-18.1  Advanced Micro Devices, Inc. [AMD] Device 141b
           +-18.2  Advanced Micro Devices, Inc. [AMD] Device 141c
           +-18.3  Advanced Micro Devices, Inc. [AMD] Device 141d
           +-18.4  Advanced Micro Devices, Inc. [AMD] Device 141e
           \-18.5  Advanced Micro Devices, Inc. [AMD] Device 141f

(In reply to Joel Schopp from comment #15)
> > (In reply to Joel Schopp from comment #10)
> > > > AMD would need to confirm it.
> > >
> > > I don't have an answer for you offhand.  Let me do some digging and get
> you
> > > an answer.
> > 
> > I am sorry if I sounded frustrated or arrogant earlier. Any update on this?
> 
> It's not clear to me which devices were being put in the same group.  Here's
> some of my notes on your lspci output

Marti, the output of 'find /sys/kernel/iommu_groups' would be useful here.  I'll try to help based on what I think is happening...

> lspci -vt
> -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1422
>            +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1423
>            +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Kaveri [Radeon R7
> 200 Series]
>            +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Device 1308
>            +-02.0  Advanced Micro Devices, Inc. [AMD] Device 1424
>            +-03.0  Advanced Micro Devices, Inc. [AMD] Device 1424
>            +-04.0  Advanced Micro Devices, Inc. [AMD] Device 1424
>            +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
>            +-10.1  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
> 
> These xhci controllers are isolated from from the other devices, I would
> need some more detail on which variant you are running to determine if they
> are isolated from eachother, they probably aren't.

10.0 & 10.1 will typically be grouped together due to lack of ACS.  This is usually not a problem.

>            +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller
> [AHCI mode]
> The sata controller is isolated from the other devices

Yep, and it's a single function device so IOMMU groups should be ok.

>            +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
>            +-12.2  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
> This pair of OHCI/EHCI controllers are together isolated from the other
> devices

Yep, same as above.

>            +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
>            +-13.2  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
> This pair of OHCI/EHCI controllers are together isolated from the other
> devices

Yep

>            +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
>            +-14.1  Advanced Micro Devices, Inc. [AMD] FCH IDE Controller
>            +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
>            +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
> I do not think the SMBus/IDE/Azalia/LPC are isolated from eachother, but
> they are isolated from the other devices I have identified.
> 
> 
>            +-14.4-[01]----05.0  Dialogic Corporation PRI
> The legacy PCI should be isolated from the other devices identified.  Not
> sure what is going on here.
> 
>            +-14.5  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
> This OHCI Controller should also be isolated from the other devices.

All of the above will be grouped together, this is the problem.  Since none of these functions support ACS, IOMMU groups assume that peer-to-peer between functions is possible.  If 14.4 and 14.5 are truly isolated from the rest of the package then we should have quirks to support that.  This whole block is an update or the quirk already shown in comment 7.

>            +-15.0-[02]--
>            +-15.2-[03]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed
> USB Host Controller
> Is this in a PCI-e slot or otherwise attached to the PCI-e?
> 
>            +-15.3-[04]----00.0  Qualcomm Atheros QCA8171 Gigabit Ethernet
> Is this in a PCI-e slot or otherwise attached to the PCI-e?

I would guess 15.x are all PCIe root ports, hopefully with ACS support.

It's an ASRock FM2A88X Extreme6+ motherboard with the AMD A88X (Bolton-D4) chipset.

There are 12 IOMMU groups on the system. The problematic group for me is number 9 because the legacy PCI bridge (14.4) gets mixed in with other southbridge devices (all 14.*).

/sys/kernel/iommu_groups/0/devices:
0000:00:00.0 -> ../../../../devices/pci0000:00/0000:00:00.0

/sys/kernel/iommu_groups/1/devices:
0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1

/sys/kernel/iommu_groups/2/devices:
0000:00:02.0 -> ../../../../devices/pci0000:00/0000:00:02.0

/sys/kernel/iommu_groups/3/devices:
0000:00:03.0 -> ../../../../devices/pci0000:00/0000:00:03.0

/sys/kernel/iommu_groups/4/devices:
0000:00:04.0 -> ../../../../devices/pci0000:00/0000:00:04.0

/sys/kernel/iommu_groups/5/devices:
0000:00:10.0 -> ../../../../devices/pci0000:00/0000:00:10.0
0000:00:10.1 -> ../../../../devices/pci0000:00/0000:00:10.1

/sys/kernel/iommu_groups/6/devices:
0000:00:11.0 -> ../../../../devices/pci0000:00/0000:00:11.0

/sys/kernel/iommu_groups/7/devices:
0000:00:12.0 -> ../../../../devices/pci0000:00/0000:00:12.0
0000:00:12.2 -> ../../../../devices/pci0000:00/0000:00:12.2

/sys/kernel/iommu_groups/8/devices:
0000:00:13.0 -> ../../../../devices/pci0000:00/0000:00:13.0
0000:00:13.2 -> ../../../../devices/pci0000:00/0000:00:13.2

/sys/kernel/iommu_groups/9/devices:
0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0
0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1
0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2
0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3
0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4
0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5
0000:01:05.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:01:05.0

    [When I plug in a card to the other legacy PCI slot, it also appears here as
     pci0000:00/0000:00:14.4/0000:01:06.0]

/sys/kernel/iommu_groups/10/devices:
0000:00:15.0 -> ../../../../devices/pci0000:00/0000:00:15.0
0000:00:15.2 -> ../../../../devices/pci0000:00/0000:00:15.2
0000:00:15.3 -> ../../../../devices/pci0000:00/0000:00:15.3
0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:15.2/0000:03:00.0
0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:15.3/0000:04:00.0

/sys/kernel/iommu_groups/11/devices:
0000:00:18.0 -> ../../../../devices/pci0000:00/0000:00:18.0
0000:00:18.1 -> ../../../../devices/pci0000:00/0000:00:18.1
0000:00:18.2 -> ../../../../devices/pci0000:00/0000:00:18.2
0000:00:18.3 -> ../../../../devices/pci0000:00/0000:00:18.3
0000:00:18.4 -> ../../../../devices/pci0000:00/0000:00:18.4
0000:00:18.5 -> ../../../../devices/pci0000:00/0000:00:18.5

(In reply to Joel Schopp from comment #15)
> It's not clear to me which devices were being put in the same group.  Here's
> some of my notes on your lspci output

Other than the 14.* devices everything seems to be as you describe.

>            +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
>            +-14.1  Advanced Micro Devices, Inc. [AMD] FCH IDE Controller
>            +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
>            +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
> I do not think the SMBus/IDE/Azalia/LPC are isolated from eachother, but
> they are isolated from the other devices I have identified.

Ok, that's not a problem.

>            +-14.4-[01]----05.0  Dialogic Corporation PRI
> The legacy PCI should be isolated from the other devices identified.  Not
> sure what is going on here.

Yep, currently shares group 9.

>            +-14.5  Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller
> This OHCI Controller should also be isolated from the other devices.

Also shares group 9.

>            +-15.0-[02]--
>            +-15.2-[03]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed
> USB Host Controller
> Is this in a PCI-e slot or otherwise attached to the PCI-e?

Nope, this is integrated on the motherboard. The only used PCI slot is the Dialogic card.

>            +-15.3-[04]----00.0  Qualcomm Atheros QCA8171 Gigabit Ethernet
> Is this in a PCI-e slot or otherwise attached to the PCI-e?

Integrated Ethernet.

The fix is now upstream and part of Linux v3.17-rc2.

(In reply to Joel Schopp from comment #15)
> It's not clear to me which devices were being put in the same group.

Hi Joel, any updates on this? I posted my IOMMU groups in comment #17 in case you missed it.

What updates are you looking for?  Joerg's fix is now upstream.

(In reply to Joel Schopp from comment #20)
> What updates are you looking for?  Joerg's fix is now upstream.

Yes, but there's still the issue with southbridge component isolation. You requested more information from me in comment #15 that I provided in comment #17.

For background see comment #9 from Alex Williamson:
> AMD would need to confirm it.  IOMMU groups are based on hardware advertised
> isolation via the PCIe ACS capability.  Without this, or a device specific
> quirk to take its place, IOMMU groups must assume that peer-to-peer between
> functions of a multi-function device is possible and therefore that the
> devices are not isolated. [...]

> I think the path forward is to get confirmation from AMD that these function
> are isolated from each other and add quirks to the kernel.  Then you won't
> have the device dependencies in vfio-pci.  The override patch allows you to
> do that with just a kernel boot parameter.  There's no gurantee that
> pci-assign will ever be fixed since it's being phased out.

Since I did not get further confirmation from Mr. Schopp, I decided to push it and submit a patch: https://lkml.org/lkml/2014/10/2/223

The phrases "Not sure what is going on here" and "should also be isolated" in comment #15 don't inspire much confidence, but I have not managed to obtain more concrete statements.

Closing bug, ACS patch merged to mainline Linux:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=3587e625fe24a2d1cd1891fc660c3313151a368c

Thanks Joerg.