Similar finding:
[    1.144340] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    1.144456] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.144457] pci 0000:00:00.2: PCI INT A: not connected
[    1.144521] pci 0000:00:01.0: Adding to iommu group 0
[    1.144774] pci 0000:00:01.2: Adding to iommu group 1
[    1.145009] pci 0000:00:01.3: Adding to iommu group 2
[    1.145245] pci 0000:00:01.6: Adding to iommu group 3
[    1.145468] pci 0000:00:08.0: Adding to iommu group 4
[    1.145677] pci 0000:00:08.1: Adding to iommu group 5
[    1.145899] pci 0000:00:08.2: Adding to iommu group 4
[    1.145924] pci 0000:00:14.0: Adding to iommu group 6
[    1.146173] pci 0000:00:14.3: Adding to iommu group 6
[    1.146224] pci 0000:00:18.0: Adding to iommu group 7
[    1.146438] pci 0000:00:18.1: Adding to iommu group 7
[    1.146464] pci 0000:00:18.2: Adding to iommu group 7
[    1.146481] pci 0000:00:18.3: Adding to iommu group 7
[    1.146498] pci 0000:00:18.4: Adding to iommu group 7
[    1.146515] pci 0000:00:18.5: Adding to iommu group 7
[    1.146542] pci 0000:00:18.6: Adding to iommu group 7
[    1.146560] pci 0000:00:18.7: Adding to iommu group 7
[    1.146598] pci 0000:01:00.0: Adding to iommu group 8
[    1.146851] pci 0000:02:00.0: Adding to iommu group 9
[    1.147143] pci 0000:03:00.0: Adding to iommu group 10
[    1.147467] pci 0000:04:00.0: Adding to iommu group 11
[    1.147606] pci 0000:04:00.0: Using iommu direct mapping
[    1.147648] pci 0000:04:00.1: Adding to iommu group 12
[    1.147875] pci 0000:04:00.2: Adding to iommu group 12
[    1.147912] pci 0000:04:00.3: Adding to iommu group 12
[    1.147941] pci 0000:04:00.4: Adding to iommu group 12
[    1.147979] pci 0000:04:00.5: Adding to iommu group 12
[    1.148015] pci 0000:04:00.6: Adding to iommu group 12
[    1.148032] pci 0000:05:00.0: Adding to iommu group 4
[    1.148310] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.148311] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    1.148312]  PPR NX GT IA GA PC GA_vAPIC

Created attachment 288095 [details]
dmesg e495

Created attachment 295035 [details]
Proposed workaround for the IOMMU PMC issue

Could you please try the attached patch to see if this help fix the issue on your platform?

Created attachment 295089 [details]
(RFCv2) Proposed workaround for the IOMMU PMC issue

RFCv2 patch adds the logic to retry after 10msec wait for each retry loop. I have founded that certain platform takes about 10msec for the power gating to disable.

Please give this a try to see if this works better on your platform.

Thanks,
Suravee

Created attachment 295091 [details]
(RFCv3) Proposed workaround for the IOMMU PMC issue

RFCv3 patch adds the logic to retry checking after 20msec wait for each retry loop since I have founded that certain platform takes about 10msec for the power gating to disable.

Please give this a try to see if this works better on your platform.

Thanks,
Suravee

Hello, Suravee,  I tried your patch in my Thinkpad E495. With RFCv1 and had no success, I'm sorry I forgot to save a dmesg for that, but I am sure it was like no patch was used, it was the same dmesg as before.

I had success with RFCv2,  and got the following:

[    0.606952] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.607117] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.607118] pci 0000:00:00.2: PCI INT A: not connected
[    0.607169] pci 0000:00:01.0: Adding to iommu group 0
[    0.607180] pci 0000:00:01.1: Adding to iommu group 1
[    0.607191] pci 0000:00:01.2: Adding to iommu group 2
[    0.607204] pci 0000:00:01.3: Adding to iommu group 3
[    0.607215] pci 0000:00:01.6: Adding to iommu group 4
[    0.607238] pci 0000:00:08.0: Adding to iommu group 5
[    0.607249] pci 0000:00:08.1: Adding to iommu group 6
[    0.607257] pci 0000:00:08.2: Adding to iommu group 5
[    0.607271] pci 0000:00:14.0: Adding to iommu group 7
[    0.607279] pci 0000:00:14.3: Adding to iommu group 7
[    0.607307] pci 0000:00:18.0: Adding to iommu group 8
[    0.607314] pci 0000:00:18.1: Adding to iommu group 8
[    0.607321] pci 0000:00:18.2: Adding to iommu group 8
[    0.607328] pci 0000:00:18.3: Adding to iommu group 8
[    0.607336] pci 0000:00:18.4: Adding to iommu group 8
[    0.607343] pci 0000:00:18.5: Adding to iommu group 8
[    0.607350] pci 0000:00:18.6: Adding to iommu group 8
[    0.607357] pci 0000:00:18.7: Adding to iommu group 8
[    0.607368] pci 0000:01:00.0: Adding to iommu group 9
[    0.607380] pci 0000:02:00.0: Adding to iommu group 10
[    0.607392] pci 0000:03:00.0: Adding to iommu group 11
[    0.607403] pci 0000:04:00.0: Adding to iommu group 12
[    0.607434] pci 0000:05:00.0: Adding to iommu group 13
[    0.607474] pci 0000:05:00.1: Adding to iommu group 14
[    0.607493] pci 0000:05:00.2: Adding to iommu group 14
[    0.607511] pci 0000:05:00.3: Adding to iommu group 14
[    0.607530] pci 0000:05:00.4: Adding to iommu group 14
[    0.607548] pci 0000:05:00.5: Adding to iommu group 14
[    0.607566] pci 0000:05:00.6: Adding to iommu group 14
[    0.607571] pci 0000:06:00.0: Adding to iommu group 5
[    0.609950] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.609951] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.609952]  PPR NX GT IA GA PC GA_vAPIC
[    0.609954] AMD-Vi: Interrupt remapping enabled
[    0.609954] AMD-Vi: Virtual APIC enabled
[    0.610822] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.611644] amd_uncore: AMD NB counters detected
[    0.611647] amd_uncore: AMD LLC counters detected
[    0.611963] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

I will attach the full dmesg later.
Should I try the v3?  
Is there anything else apart from booting and checking dmesg that I can do to test your patch?

Created attachment 295199 [details]
Thinkpad E495 with RFCv2 patch.

RFC v2 and v3 should be similar. I have modified the retry loop a bit in V3 to be more efficient.

Patch has been submitted here (https://lkml.org/lkml/2021/2/8/486)

Thanks,
Suravee

Created attachment 295423 [details]
Linux 5.12+ messages

Some days ago, Linus added the commit to this main branch [1].

Unfortunately, I am still seeing this error.

    [    0.000000] Linux version 5.11.0-10281-g19b4f3edd5c9 (root@a2ab663d937e) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.1) #138 SMP Wed Feb 24 11:28:17 UTC 2021
    […]
    [    0.106422] smpboot: CPU0: AMD Ryzen 3 2200G with Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0)
    […]
    [    0.291257] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
    […]

@00oo00, is the error gone for you now (AMD Ryzen 5 2400G with Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0))?


[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6778ff5b21bd8e78c8bd547fd66437cf2657fd9b

Hi Suravee! 

This bug has been bothering Ryzen users for some while now and work-arounds adding various linux boot parameters have had patchy success. It's good to see an informed and more forensic approach. Bravo!

I've been exercising your RFCv3 patch over the last few days (AMD Ryzen 5 2400G, Asus Prime B350-Plus, Ubuntu 20.10, Linux 5.8.0-43). With the original 20 msec delay I still get:

[    0.703538] Trying to unpack rootfs image as initramfs...
[    0.859160] Freeing initrd memory: 83748K
[    0.859256] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.859265] fbcon: Taking over console
[    0.859367] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.859368] pci 0000:00:00.2: PCI INT A: not connected
[    0.859413] pci 0000:00:01.0: Adding to iommu group 0
                                                      ...
                                                      ...
[    0.859959] pci 0000:0b:00.0: Adding to iommu group 12
[    0.863140] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.863142] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.863143]  PPR NX GT IA GA PC GA_vAPIC
[    0.863144] AMD-Vi: Interrupt remapping enabled
[    0.863145] AMD-Vi: Virtual APIC enabled
[    0.863375] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.864261] amd_uncore: AMD NB counters detected
[    0.864273] amd_uncore: AMD LLC counters detected
[    0.864696] check: Scanning for low memory corruption every 60 seconds

Increasing the delay to msleep(40) gives:

[    0.212761] Trying to unpack rootfs image as initramfs...
[    0.366407] Freeing initrd memory: 83304K
[    0.509105] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.509199] pci 0000:00:00.2: can't derive routing for PCI INT A
[    0.509200] pci 0000:00:00.2: PCI INT A: not connected
[    0.509244] pci 0000:00:01.0: Adding to iommu group 0
                                                      ...
                                                      ...
[    0.509802] pci 0000:0b:00.0: Adding to iommu group 12
[    0.512997] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.512999] pci 0000:00:00.2: AMD-Vi: Extended features (0x4f77ef22294ada):
[    0.512999]  PPR NX GT IA GA PC GA_vAPIC
[    0.513001] AMD-Vi: Interrupt remapping enabled
[    0.513001] AMD-Vi: Virtual APIC enabled
[    0.513181] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.514116] amd_uncore: AMD NB counters detected
[    0.514128] amd_uncore: AMD LLC counters detected
[    0.514482] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.514534] check: Scanning for low memory corruption every 60 seconds

If Linus is taking up the patch (hurrah), is 40 msec too large over the Ryzen range? I would quite like the good Ubuntu people to merge your patch into their current and upcoming distributions (I think SuSE are already doing so) but don't want to jump-the-gun or tread-on-any-toes :-).

Over the next few days, I'll try my 2400G on delays between 20 and 40 msec and maybe even test the patch on the still rather unstable Ubuntu 21.04. Any guidance
much appreciated and, once again, many thanks.

David

Good morning again!

I've rebuilt the Ubuntu 20.10 kernel 5.8.0-44 overnight with increased delays. Grep'ing dmesg for IOMMU gives:

20 msec delay:

[    0.506947] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.510822] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.764160] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

30 & 40 msec delay:

[    1.017676] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.021591] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.023053] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.345196] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

At least on my 2400G platform, that suggests 20 msec is still too short. Of course, YMMV :-).

Machines are grinding through the same trial on Ubuntu 21.04 alpha's 5.10.11 kernel and I'll post results in due course.

Best regards

David

@David, please increase the number of retries instead of increasing the delay. Please also print out the number of retries.

Looking at the patch, delaying the boot by up to 100 ms, and no proper logging, it leaves a lot to be desired.

@David, you can also test Alex’ patch *iommu/amd: Fix event counter availability check* [1].


[1]: https://lore.kernel.org/linux-iommu/20200529200738.1923-1-amonakov@ispras.ru/

An alternative solution to remove the writes is suggested in [1].

[1]: https://lore.kernel.org/linux-iommu/alpine.LNX.2.20.13.2006030935570.3181@monopod.intra.ispras.ru/

Good evening, all!

Two extra results from the Ryzen 2400G:

1. 5 x 25 msec loop on Ubuntu 21.04, kernel 5.10.11 and Suravee's original patch.  Works fine!

2. 10x 20 msec loop on Ubuntu 20.10, kernel 5.8.18 with both Suravee and Alex's patches. Also fine!

   As expected,
      AMD-Vi: IOMMU performance counters supported
   moves to after
      Adding to iommu group ...

This means that, for this CPU, the power-gating needs between 100 - 125 msec to disable. A logging printf would have got me there faster (not overly fond of kernel rebuilds) but the missing information is the maximum settling-time needed across the full Ryzen range. Paul's suggestion of increasing the loop count to 10 gives 200 msec maximum headroom. Surely must be enough?

Now, all we need is a work-around for the PCI INT A: not connected diagnostic :-).

Many thanks good people. Keep safe.

David

Thank you for the test. Please do not mix Suravee’s and Alex’ patches. I think it should be either or.

I am going to respond on the list, but in my opinion, it’s not an option to add delays and hold up the fast boot up further.

Aaah! Should have read Alex's text properly :-(.

Again, good day!

Two re-builds using Alex's (one-line) patch, for kernels 5.8.18 and 5.10.11. Both give on the Ryzen 2400G:

[    0.374138] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.374141] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.375457] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.626269] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

I don't know how significant an extra 125 msec on boot time is but, on grounds of simplicity and elegance, I'd vote for Alex :-). Are there any downsides?

Much appreciated all.

David

Even with ten tries in the loop, it still fails with the AMD Ryzen 3 2200G with Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0):

    [    0.401152] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter. (retry = 0)

```
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 9126efcbaf2c7..70c00ee5ff354 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1746,7 +1746,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu *iommu)
 
        /* Check if the performance counters can be written to */
        val = 0xabcd;
-       for (retry = 5; retry; retry--) {
+       for (retry = 10; retry; retry--) {
                if (iommu_pc_get_set_reg(iommu, 0, 0, 0, &val, true) ||
                    iommu_pc_get_set_reg(iommu, 0, 0, 0, &val2, false) ||
                    val2)
@@ -1764,7 +1764,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu *iommu)
        if (val != val2)
                goto pc_false;
 
-       pci_info(pdev, "IOMMU performance counters supported\n");
+       pci_info(pdev, "IOMMU performance counters supported (retry = %i)\n", retry);
 
        val = readl(iommu->mmio_base + MMIO_CNTR_CONF_OFFSET);
        iommu->max_banks = (u8) ((val >> 12) & 0x3f);
@@ -1773,7 +1773,7 @@ static void __init init_iommu_perf_ctr(struct amd_iommu *iommu)
        return;
 
 pc_false:
-       pci_err(pdev, "Unable to read/write to IOMMU perf counter.\n");
+       pci_err(pdev, "Unable to read/write to IOMMU perf counter. (retry = %i)\n", retry);
        amd_iommu_pc_present = false;
        return;
 }
```

Hi all!

It's a knotty problem: the Ryzen has IOMMU performance counters, but can't tell you they're working without a (120 msec 2400g, > 200 msec 2200G) delay (not popular with Paul or Alex). It can assume they are working pro-tem but check later in the boot sequence (Alex's patch but Suravee has some products that don't like this). It can assume they aren't working (although they are) and leave the user-space application to handle the loss of facility (not at all keen on this myself).

System OEM's will want clean boot interfaces and road-warriors with high-end laptops hate unnecessary delay when they lift their lids. However, day-to-day users of virtualisation software like (for example) KDM will notice if IOMMU/Vi hardware acceleration is present but crippled by the OS (how do the Windows drivers handle the same issue?). The usual solution for this sort of dilemma, I think, is to put in a BIOS switch or have a boot command-line option.

The current default is to turn slow-reporting IOMMU performance counters off. This gives problems later with:

# perf stat -a -e amd_iommu/mem_trans_total/ test
event syntax error: 'amd_iommu/mem_trans_total/'
                     \___ Cannot find PMU `amd_iommu'. Missing kernel support?

After either Suravee or Alex's patch:

# perf stat -a -e amd_iommu/mem_trans_total/ test
Performance counter stats for 'system wide':
               897      amd_iommu/mem_trans_total/                               
       0.000990106 seconds time elapsed

Installation checks for KVM like systool -m kvm_amd -v and kvm-ok then give expected results about modules and hardware acceleration. In principle, I'm sorted for the 2400G but repeatedly recompiling the kernel is not a lovely solution. Very, very appreciative of all attempts to get a generally-acceptable patch into the distributions!

Best regards

David

More results and some progress!!

A. Alex Hung (Ubuntu) has cherry-picked the kernel-commit [1} of Suravee's version 3 patch and placed test-builds on [2]. It works for his Ryzen 2500U.  Bravo for a quick take-up!

I've checked it out and (alas, as expected with the original 5 x 20 msec delay) it is isn't enough for my Ryzen 2400G which needs ~120 msecs to respond :-(.

B. I've done a couple more rebuilds for the current kernels on Ubuntu 20.10 and 21.04 using Suravee's patch updated both with Pauls' retry-logging and with increasing the maximum delay to 25 x 20 msec.

Grepp'ing dmesg for IOMMU (on my 2400g) gives:

[    1.018329] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported (retry = 19)
[    1.022185] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.023669] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.283218] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

The retrys count backwards so that confirms the Ryzen 2500G still needs 120 msecs to respond.

My (very strong) proposal is that we increase Suravee's retry-count to something large enough to cover the numerous entry-level Ryzen's out there. Only the absolutely necessary boot-up delay is introduced (and checked, if required, by Paul's logging statement). Most purchasers of Ryzen CPUs are then covered! Only the dyed-in-the-wool speed-merchants have to make a choice between boot-up time and IOMMU capability - at least until the underlying problem can be corrected in firmware or silicon.

Thanks again, good people.

David

[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6778ff5b21bd8e78c8bd547fd66437cf2657fd9b
[2]. https://people.canonical.com/~alexhung/LP1917203/

Good morning, Suravee!

One (final?!) test of your patch (with increased retry count and logging) on the latest git kernel (5.10.0-rc2) all on Ryzen 2400G. In summary:

 5.8.0-44   (Ubuntu 20.10) 6 trys 20 msecs
 5.10.0-14  (Ubuntu 21.04) 5 trys 20 msecs
 5.12.0-rc2 (raw git)      6 trys 20 msecs

AFAIK you are AMD's go-to guru for open-source people and an IOMMU expert. It's your patch that is now both in the kernel commit and (I hope) in Ubuntu's to-do list. Could I ask if you could patch-the-patch (if there is no esoteric technical downside) to increase the number of retrys? Maybe if Paul would give it another shot with his Ryzen 2200G with a larger maximum count and his logging :-).

Best regards and our thanks

David

Just a note, maybe useless.
I own a Huawei laptop, model KPL-W00D with Ryzen 2500U and it suffer this same bug.

I cannot compile a kernel so, after installing Devuan testing, I installed also the generic kernel located here:
https://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2021-03-15/amd64/
that is a 5.12.0 RC3 and it includes the upstream patch for this bug.

The patch works but only on cold boots, after a simple reboot the bug turns back again.

Maybe it's only a hardware problem, maybe a distro fault or misconfiguration, I don't know, but I felt compelled to report this.

If needed I can give more detailed informations.

I would guess that the difference between warm and cold boot is due to your Ryzen 2500U (like my 2440G) hovering between needing 5 and 6 tries of 20 msecs. Without adding Paul's logging you can't tell!

If it would help anyone (and you trust a non-official kernel recompile) I have placed 4 variants on my Box account. Both Suravee's original patch with 25 x 20 msec tries maximum and logged and Alex's very simple patch are applied to Ubuntu's 5.8.0-44 (20.10) and 5.10.0-14 (21.04) kernels.

The version numbers are identical so that Ubuntu's own DEB's for linux-tools, linux-headers and linux-libc-dev are compatible. My linux-image replaces their own linux-image and linux-modules so take care that you can still boot from a 'known' kernel.

They all work for me with quite heavy use on KVM but YMMV :-). Mail me if you need access.

Best regards to all and my thanks to Suravee, Alex and Paul!

David

Just confirmed BubuXP's Ryzen 2500U result for Suravee's (logged) patch on my 2400G with the latest Ubuntu 5.8.0-45 kernel. Cold boot is marginally faster, needing 5 x 20 msec, rather than 6 x 20 msecs for warm boot.

David

I do not know if this will help your testing, but I am using MSI 450-A Pro and AMD Ryzen 3 2200G with Radeon Vega Graphics.  When booting into Debian 10 I have always seen the error message that states:
"Unable to write to  IOMMU perf counter"
It flashes at the top of the screen as the first item then goes away only to be rewritten at the top of the boot process steps.  It only last a moment and then everything boots normally.

However, I just updated to the latest BIOS version and it hung at that point every time.  The BIOS version was Beta, so re-flashing to the older version solved the problem.  It is now back to its normal behavior.

I'm not into manual kernel changes or configuring from source, so I'm sorry if I can't help more.

Hope this info helps your research, and thanks for it.  Keep up the great work.

(In reply to kitchm from comment #25)
> I do not know if this will help your testing, […]

Thank you for your comment. This seems like a firmware issue. Please report that to MSI, so they can fix that regression in their stable(?) release.

Lastly, please always add the Linux kernel version (`/proc/version`), and the exact firmware version numbers (`sudo dmesg | grep DMI`).

Good morning,
I'm using the latest 5.11.7 kernel on an Aura15 Tuxedo notebook (based on Amd Ryzen 4700U). As far as I can tell, the patch was merged on 5.11.7, but at startup (cold boot) I read:

[    1.888186] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    1.888454] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.888457] pci 0000:00:00.2: PCI INT A: not connected
[    1.888510] pci 0000:00:01.0: Adding to iommu group 0
[    1.888537] pci 0000:00:01.3: Adding to iommu group 1
[    1.888565] pci 0000:00:02.0: Adding to iommu group 2
[    1.888583] pci 0000:00:02.1: Adding to iommu group 3
[    1.888600] pci 0000:00:02.2: Adding to iommu group 4
[    1.888616] pci 0000:00:02.4: Adding to iommu group 5
[    1.888644] pci 0000:00:08.0: Adding to iommu group 6
[    1.888663] pci 0000:00:08.1: Adding to iommu group 6
[    1.888689] pci 0000:00:14.0: Adding to iommu group 7
[    1.888707] pci 0000:00:14.3: Adding to iommu group 7
[    1.888764] pci 0000:00:18.0: Adding to iommu group 8
[    1.888779] pci 0000:00:18.1: Adding to iommu group 8
[    1.888795] pci 0000:00:18.2: Adding to iommu group 8
[    1.888810] pci 0000:00:18.3: Adding to iommu group 8
[    1.888825] pci 0000:00:18.4: Adding to iommu group 8
[    1.888840] pci 0000:00:18.5: Adding to iommu group 8
[    1.888855] pci 0000:00:18.6: Adding to iommu group 8
[    1.888869] pci 0000:00:18.7: Adding to iommu group 8
[    1.888886] pci 0000:01:00.0: Adding to iommu group 9
[    1.888906] pci 0000:02:00.0: Adding to iommu group 10
[    1.888923] pci 0000:03:00.0: Adding to iommu group 11
[    1.888940] pci 0000:04:00.0: Adding to iommu group 12
[    1.888959] pci 0000:05:00.0: Adding to iommu group 6
[    1.888968] pci 0000:05:00.1: Adding to iommu group 6
[    1.888977] pci 0000:05:00.2: Adding to iommu group 6
[    1.888985] pci 0000:05:00.3: Adding to iommu group 6
[    1.888993] pci 0000:05:00.4: Adding to iommu group 6
[    1.889000] pci 0000:05:00.5: Adding to iommu group 6
[    1.889008] pci 0000:05:00.6: Adding to iommu group 6
[    1.891539] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.891545] pci 0000:00:00.2: AMD-Vi: Extended features (0x206d73ef22254ade):
[    1.891547]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    1.891552] AMD-Vi: Interrupt remapping enabled
[    1.891552] AMD-Vi: Virtual APIC enabled
[    1.891553] AMD-Vi: X2APIC enabled
[    1.891781] AMD-Vi: Lazy IO/TLB flushing enabled

Thanks, Paul.  Of course you are correct.  A sloppy mistake on my part.

Linux version 4.19.0-14-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)

[    0.000000] DMI: Micro-Star International Co., Ltd MS-7B86/B450-A PRO (MS-7B86), BIOS A.D0 06/11/2020
[    0.132574] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    5.445162] input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:08.1/0000:27:00.1/sound/card0/input16
[    5.445197] input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:08.1/0000:27:00.1/sound/card0/input17

I tried to report it to MSI, but they have a terrible communication system.  Aw, well....

Thanks again.

Not fixed for me either.

I'm using a Huawei MateBook D 14 AMD Ryzen 3500u. 5.11.7 kernel on Void Linux. Same on cold boot.

Hello, a little update to my previous message: same machine, same kernel, but this time the patch works. So it seems that also on the same HW the timings are not so predictable...

[    1.844903] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.845157] pci 0000:00:00.2: can't derive routing for PCI INT A
[    1.845160] pci 0000:00:00.2: PCI INT A: not connected
[    1.845213] pci 0000:00:01.0: Adding to iommu group 0
[    1.845240] pci 0000:00:01.3: Adding to iommu group 1
[    1.845267] pci 0000:00:02.0: Adding to iommu group 2
[    1.845284] pci 0000:00:02.1: Adding to iommu group 3
[    1.845301] pci 0000:00:02.2: Adding to iommu group 4
[    1.845318] pci 0000:00:02.4: Adding to iommu group 5
[    1.845349] pci 0000:00:08.0: Adding to iommu group 6
[    1.845368] pci 0000:00:08.1: Adding to iommu group 6
[    1.845394] pci 0000:00:14.0: Adding to iommu group 7
[    1.845411] pci 0000:00:14.3: Adding to iommu group 7
[    1.845469] pci 0000:00:18.0: Adding to iommu group 8
[    1.845484] pci 0000:00:18.1: Adding to iommu group 8
[    1.845500] pci 0000:00:18.2: Adding to iommu group 8
[    1.845515] pci 0000:00:18.3: Adding to iommu group 8
[    1.845530] pci 0000:00:18.4: Adding to iommu group 8
[    1.845545] pci 0000:00:18.5: Adding to iommu group 8
[    1.845559] pci 0000:00:18.6: Adding to iommu group 8
[    1.845574] pci 0000:00:18.7: Adding to iommu group 8
[    1.845592] pci 0000:01:00.0: Adding to iommu group 9
[    1.845611] pci 0000:02:00.0: Adding to iommu group 10
[    1.845629] pci 0000:03:00.0: Adding to iommu group 11
[    1.845646] pci 0000:04:00.0: Adding to iommu group 12
[    1.845665] pci 0000:05:00.0: Adding to iommu group 6
[    1.845674] pci 0000:05:00.1: Adding to iommu group 6
[    1.845683] pci 0000:05:00.2: Adding to iommu group 6
[    1.845691] pci 0000:05:00.3: Adding to iommu group 6
[    1.845699] pci 0000:05:00.4: Adding to iommu group 6
[    1.845706] pci 0000:05:00.5: Adding to iommu group 6
[    1.845714] pci 0000:05:00.6: Adding to iommu group 6
[    1.848251] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.848257] pci 0000:00:00.2: AMD-Vi: Extended features (0x206d73ef22254ade):
[    1.848258]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    1.848263] AMD-Vi: Interrupt remapping enabled
[    1.848264] AMD-Vi: Virtual APIC enabled
[    1.848264] AMD-Vi: X2APIC enabled
[    1.848490] AMD-Vi: Lazy IO/TLB flushing enabled

Hah.  It's true,  cold booted and IOMMU was implemented, but after the second reboot, I got:

[    0.857880] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.

my system: linux 5.10.23-200.fc33.x86_64,  thinkpad e495 / AMD Ryzen 3500u

I'm pretty sure this is because IOMMU (for some Ryzens) unlatches after 5 tries on cold boot and 6 tries on warm. Suravee, alas, has set his patch at 5 x 20 msecs just on the 'dither' point!

Some time mid-next week I'll lay my sticky hands on a 4700U laptop. I have images re-compiled for Ubuntu's kernel 5.8.0-45 (20.10) and 5.11.0-11 (21.04) patched at 25 x 20 msecs and logged. Ditto with Alex's oh-so-simple patch.

If all goes well(!), I'll draw up a nice, comparative table of mine and everybody else's data so we can draw some conclusions. If you scan the Debian/Ubuntu kernel changelog, you'll find a long list of 'cherry-picked' patches for both AMD and Intel for IOMMU. It is very much a work-in-progress still!

Best regards and many thanks all for input.

David

Good evening all!

Herewith a summary of our various results, including my latest for the Ryzen 7 4700U (Acer Swift 3).

20 msec wait with logged retrys

Ryzen     Kernel            Cold    Warm

4700U   5.11.0-11            6      1
3500U   5.11.7               5      6
2500U   5.8.0-45                    5
        5.12.0 RC3           5    > 5
2400G   5.11.0-11            6      6
        5.8.0-45             5      6
2200G       ?                     >10

Two points are clear:

1. there are differences between cold and warm boot, mostly marginal but marked and very consistent with my 4700U.

2. the choice of 5 as the maximum retry number is (sorry, Suravee) unfortunate. Mostly, it guarantees that all our Ryzens just fail the IOMMU write test!

Paul's experiences with the 2200G are the exception and I can appreciate his unease about just increasing the number of retrys. I'm sure he, Suravee, Alex and the AMD engineers will find an elegant solution to the problem :-).

In the meantime, could you formally patch-the-patch, Suravee? I'll pass on the above data to Alex Hung, our "cherry-picker" at Ubuntu.

Best regards

David

I'd also like to report this issue with my Ryzen 7 4750g on my B450 based motherboard. I'm on linux kernel version 5.11.10.

I'll give log files and configs once I get a chance.

Commit 994d6608 (iommu/amd: Remove performance counter pre-initialization test) [1] fixes this, and was added to Linux 5.13.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=994d6608efe4a4c8834bdc5014c86f4bc6aceea6

@00oo00, can you please close the issue?

Whoops,  I forgot to report as well that this is fixed for me.
Thinkpad E495 / Ryzen 3500u / debian 11 linux 5.10.46-5


[    0.366749] iommu: Default domain type: Translated 
[    0.805580] pci 0000:00:01.0: Adding to iommu group 0
[    0.805589] pci 0000:00:01.1: Adding to iommu group 1
[    0.805600] pci 0000:00:01.2: Adding to iommu group 2
[    0.805608] pci 0000:00:01.3: Adding to iommu group 3
[    0.805616] pci 0000:00:01.6: Adding to iommu group 4
[    0.805635] pci 0000:00:08.0: Adding to iommu group 5
[    0.805644] pci 0000:00:08.1: Adding to iommu group 6
[    0.805651] pci 0000:00:08.2: Adding to iommu group 5
[    0.805663] pci 0000:00:14.0: Adding to iommu group 7
[    0.805670] pci 0000:00:14.3: Adding to iommu group 7
[    0.805701] pci 0000:00:18.0: Adding to iommu group 8
[    0.805709] pci 0000:00:18.1: Adding to iommu group 8
[    0.805716] pci 0000:00:18.2: Adding to iommu group 8
[    0.805723] pci 0000:00:18.3: Adding to iommu group 8
[    0.805730] pci 0000:00:18.4: Adding to iommu group 8
[    0.805736] pci 0000:00:18.5: Adding to iommu group 8
[    0.805743] pci 0000:00:18.6: Adding to iommu group 8
[    0.805749] pci 0000:00:18.7: Adding to iommu group 8
[    0.805758] pci 0000:01:00.0: Adding to iommu group 9
[    0.805767] pci 0000:02:00.0: Adding to iommu group 10
[    0.805775] pci 0000:03:00.0: Adding to iommu group 11
[    0.805783] pci 0000:04:00.0: Adding to iommu group 12
[    0.805809] pci 0000:05:00.0: Adding to iommu group 13
[    0.805835] pci 0000:05:00.1: Adding to iommu group 14
[    0.805846] pci 0000:05:00.2: Adding to iommu group 14
[    0.805857] pci 0000:05:00.3: Adding to iommu group 14
[    0.805868] pci 0000:05:00.4: Adding to iommu group 14
[    0.805879] pci 0000:05:00.5: Adding to iommu group 14
[    0.805890] pci 0000:05:00.6: Adding to iommu group 14
[    0.805895] pci 0000:06:00.0: Adding to iommu group 5
[    0.810116] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).