I just hit exactly the same issue when upgrading the kernel from v5.13.0-40 to v5.15.0-27. With no devices plugged in, the USB hub reports everything as ok. Plugging in a USB keyboard worked for a minute or two, and then I get exactly the same errors from [+0,229229] to [+0,000004] above.

Same USB controller chipset as OP by the looks of things. I've managed to list the Capabilities in case that's any help:
03:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller
        Flags: bus master, fast devsel, latency 0, IRQ 28, IOMMU group 12
        Memory at e0a00000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

Downgrading back to v5.13.0-40 fixes the problem.

I wanted to add this issue to the regression tracking and poke the maintainers, but noticed there is a patch that is being backported right now that might or might not be related (not my area of expertise):
https://lore.kernel.org/all/20220504153117.726462014@linuxfoundation.org/

It's already in 5.18-rc5; could somebody please give it a quick try before I proceed with my initial plan?

I built myself this kernel 5.18-rc5 (with ubuntu default config), but the problem still exists

I've misled you a bit by saying that the bug didn't occur on the 5.13 kernel. I tried bisecting on the upstream kernel and it turns out that the problem also occurs on the 5.13 - I build it using ubuntu default config from kernel 5.15.0-27.

So far, the only kernel build I haven't noticed a problem with (excluding kernels 5.4 from Ubuntu 20.04 LTS) is kernel 5.13.0-28-generic form Ubuntu.

Interestingly, I found the sources of this kernel on git kernel.ubuntu.com and built this kernel using this config from kernel 5.15 and the problem also occurred.

It was only when I built this kernel using the default config for this kernel that I stopped seeing the problem.

Sorry, this is starting to get confusing and hard to follow.

If there is something that used to work with an Ubuntu kernel and stops working there, you might want to report it to the Ubuntu developers, but not here.

This bug tracker care mainly about upstream kernel (see front page), so what happens with a kernel build from the Ubuntu sources (which are known to be modified a lot) is irrelevant and even just mentioning that makes things hard to follow. :-/

Regarding your problem: I'm not familiar with the code that might cause this, but to me it looks a lot like Ubuntu switched on a kernel configuration option that is causing this. If that's the case the problem doesn't qualify as regression, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

The developers nevertheless might be interested in fixing this, but might need more details from you (like the config option that is causing this)

[  +0,229229] xhci_hcd 0000:04:00.0: WARNING: Host System Error

The xHC controller reports a catastrophic error, and sets HSE bit.

For PCI xHC controllers the spec lists possible causes as:
host controller PCI parity error, PCI Master Abort, PCI Target Abort.
But DMA issues also possible cause, especially as log shows  DMAR
problems right after this.

Any chance you could bisect this on upstream kernel?

@Thorsten Leemhuis sorry for misleading you but when adding this bug here, I didn't know it wasn't an upstream regression - at first look it looked that way, as I also observed the problem on the upstream.

So far we only know that in one of the kernel configurations the problem does not occur - but this does not mean that the problem does not exist.

> Any chance you could bisect this on upstream kernel?

I'll try to do it this week

(In reply to Piotr Piórkowski from comment #7)
>
> > Any chance you could bisect this on upstream kernel?
> I'll try to do it this week

And news? Was the issue maybe fixed meanwhile?

The IOMMU error is caused by a buggy VL805 firmware. It is more visible with the Debian kernel as Debian patches the kernel to enable IOMMU by default. The updated firmware can be installed using the VIA Windows tool (this did not work for me), or you can just turn off IOMMU.

(In reply to Chris Bainbridge from comment #9)
> The IOMMU error is caused by a buggy VL805 firmware. 

Makes me wonder: would it be possible to detect an old firmware and avoid the IOMMU path in this case? Or at least warn?

(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #10)
> (In reply to Chris Bainbridge from comment #9)
> > The IOMMU error is caused by a buggy VL805 firmware. 
> 
> Makes me wonder: would it be possible to detect an old firmware and avoid
> the IOMMU path in this case? Or at least warn?

It would be possible to detect, the firmware version can be read with: 

$ sudo lspci -d 1106:3483 -xxx | awk '/^50:/ { print "VL805 FW version: " $5 $4 $3 $2 }'
VL805 FW version: 00013500

imho it would be a good idea for Linux to track the latest firmware versions for *all* hardware and warn if a firmware is out-of-date (even if the firmware updater is only available on Windows). Earlier this year I had an intermittent issue with a new laptop where the desktop would hang and processes would get IO errors. But this only happened once every 3 weeks or so. It took a few months to isolate the problem to NVME firmware (it was a HP laptop with Intel NVME, and I was unaware that these drives have locked HP-specific firmware). The firmware update was a Windows executable. I've also seen many forum posts where people have problems that were resolved by updates to GPU/motherboard/NVME/ethernet/wifi etc. firmware. Many of these problems could have been resolved a lot quicker if the kernel log contained "old firmware detected!".

(In reply to Chris Bainbridge from comment #11)
> It would be possible to detect […]

Thx for that.

> imho it would be a good idea for Linux to track the latest firmware versions
> for *all* hardware […]

Pretty sure that is something that should be done in userspace, as there it's a lot easier to update the dataset with the latest firmware versions.

Ideally this should be fixed, but we are kinda stuck here:

* Piotr, do you even care about this after all this time?

* does the problem even still happen with the latest mainline kernel? might be a good idea to test this with 6.6 or 6.7-rc1 is out (e.g. in two weeks from now) before doing anything else.

* The regression was never bisected, hence it's unclear which developers is resposible for handling this (USB? IOMMU? something else?). But well, with a bit of luck Mathias commented earlier and might see this and share his thoughts.

Honestly, I had already forgotten about this problem. In the meantime, I changed the HW. Lately I have little time, but I have somewhere this HW still and if there is a need I can verify something

(In reply to Piotr Piórkowski from comment #14)
> Honestly, I had already forgotten about this problem.

Happens, no worries :-D

> Lately I have little time, but I have somewhere this HW
> still and if there is a need I can verify something

I suggest you wait for Mathias to speak up first, I guess he should know best what's the best way forward here.

> Pretty sure that is something that should be done in userspace, as there it's
> a lot easier to update the dataset with the latest firmware versions.

True, but then you have the problem of convincing distributions to ship it and enable it by default.

> The regression was never bisected

It's not a regression in the kernel as Piotr said in comment #7. It appeared on Debian kernels because Debian enabled IOMMU by default (but not for the GPU). This is a Debian-specific patch.

If I understand correctly this is caused by the VIA VL805 xHC controller with bad firmware accessing some DMA address outside the allowed range.

With IOMMU enabled the IOMMU will prevent this access, and the controller fails.

I'm speculating here, but it could be possible the controller accesses past one of the DMA ranges wile trying to read-ahead.

If we can figure out past which area, then its possible to make a driver workaround for this controller that allocates a bit larger DMA chunk for that specific purpose. 

DMA memory allocated for xHC use before any USB device is connected:

- dcbaa device context base address array.
  arrays of pointers to device contexts
  dma_alloc_coherent(dev)

- command ring
  dma_pool_zalloc(segment_pool)

- event ring
  dma_pool_zalloc(segment_pool)

- event ring segment table
  info about event ring, segents, size and location
  dma_alloc_coherent(dev)

- Scratchpad,
  only touched (RW) by xHC controller, not driver.
  dma_alloc_coheret()

DMA memory allocated for each connected USB device.

- device contexts
  dma_pool_zalloc(device_pool)

- transfer rings,
  contains TRBs, metadata about transfers.
  dma_pool_zalloc(segment_pool)

- stream contexts,
  dma_alloc_coherent() or dma_pool_alloc(*_streams_pool)

(In reply to Chris Bainbridge from comment #16)
> It's not a regression in the kernel as Piotr said in comment #7.

Ahh, sorry, missed/forgot that in all the regressions I deal with. Thx for pointing it out!

BTW, thx Mathias for nevertheless looking into this.

(In reply to Mathias Nyman from comment #17)
> If we can figure out past which area, then its possible to make a driver
> workaround for this controller that allocates a bit larger DMA chunk for
> that specific purpose. 

I am a debian user and opened this bug report:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050352

In the last mail I was pointed here.

So I now offer my help because I have the hardware and can reproduce the bug.

I am not a kernel developer. But I am able to modify kernel source files and compile a custom kernel and let it run on debian, either debian stable or debian testing.

May be I could test new versions of the driver.

By a complete accident I found that VL805 has a known bug where it overfetches transfer rings:
https://github.com/raspberrypi/linux/issues/4685

Rolf, there is a workaround for similar bug in Zhaoxin hardware which may solve your IOMMU faults.

Update to at least v6.6, or the latest v6.12 series if it's not a problem.

Run the following (if using a USB keyboard put it all in a script):
rmmod xhci_pci
rmmod xhci_hcd
modprobe xhci_hcd quirks=0x200000000000
modprobe xhci_pci

Verify that it worked:
dmesg |grep quirks
[122123.422469] xhci_hcd 0000:08:00.0: hcc params 0x002841eb hci version 0x100 quirks 0x0000200000000890
                                                                                            ^-- here
And see if it helps.

(In reply to Michał Pecio from comment #20)
> By a complete accident I found that VL805 has a known bug where it
> overfetches transfer rings:
> https://github.com/raspberrypi/linux/issues/4685
> 
> Rolf, there is a workaround for similar bug in Zhaoxin hardware which may
> solve your IOMMU faults.
> 
> Update to at least v6.6, or the latest v6.12 series if it's not a problem.
> 
> Run the following (if using a USB keyboard put it all in a script):
> rmmod xhci_pci
> rmmod xhci_hcd
> modprobe xhci_hcd quirks=0x200000000000
> modprobe xhci_pci
> 
> Verify that it worked:
> dmesg |grep quirks
> [122123.422469] xhci_hcd 0000:08:00.0: hcc params 0x002841eb hci version
> 0x100 quirks 0x0000200000000890
>                                                                             
> ^-- here
> And see if it helps.

Thank you very much.

I just tried it and it works. The USB port runs fine.

But what is actually done here?
modprobe xhci_hcd quirks=0x200000000000
A setting is given to the driver when starting, but what does this setting mean?

That's good to hear. Just to be sure, can you confirm that without this quirk the problem still exists today, on the same machine running the same kernel?

This enables the XHCI_ZHAOXIN_TRB_FETCH quirk which allocates 8KB for each 4KB transfer ring segment and leaves the upper 4KB empty, so that DMA accesses beyond the 4KB segment end up hitting valid and harmless memory.

It wastes a few KB of memory per endpoint, and RasPI came up with a more efficient solution, but this is something already implemented here so we could try it immediately without patching.

BTW, there is speculation that the Zhaoxin bug may be caused by odd page size requirements of some controllers. If your kernel has debugfs enabled, please also post the output of:

grep PAGESIZE /sys/kernel/debug/usb/xhci/*/reg-op
lspci -nn |grep USB

(In reply to Michał Pecio from comment #22)
> That's good to hear. Just to be sure, can you confirm that without this
> quirk the problem still exists today, on the same machine running the same
> kernel?

The problem still exists today.
rolf@i7-5820K-debian-testing1:~$ uname -a
Linux i7-5820K-debian-testing1 6.12.5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.5-1 (2024-12-15) x86_64 GNU/Linux

 
> BTW, there is speculation that the Zhaoxin bug may be caused by odd page
> size requirements of some controllers. If your kernel has debugfs enabled,
> please also post the output of:
> 
> grep PAGESIZE /sys/kernel/debug/usb/xhci/*/reg-op

rolf@i7-5820K-debian-testing1:~$ sudo grep PAGESIZE /sys/kernel/debug/usb/xhci/*/reg-op
grep: /sys/kernel/debug/usb/xhci/*/reg-op: Datei oder Verzeichnis nicht gefunden

Which is translated: File or directory not found


> lspci -nn |grep USB

rolf@i7-5820K-debian-testing1:~$ lspci -nn |grep USB
00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
05:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller [1106:3483] (rev 01)
06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]

Maybe debugfs is not mounted. Try
mount -t debugfs none /sys/kernel/debug

(In reply to Michał Pecio from comment #24)
> Maybe debugfs is not mounted. Try
> mount -t debugfs none /sys/kernel/debug

root@i7-5820K-debian-testing1:/sys/kernel/debug/usb/xhci/0000:05:00.0# ls -l reg-op 
-r--r--r-- 1 root root 0 23. Dez 13:10 reg-op

The file is empty.

Honestly, I don't know what's the problem. Maybe as a last ditch effort try dynamic debug, if your kernel supports it:

echo 'func xhci_dbg_trace +p' >/proc/dynamic_debug/control
rmmod xhci-pci
modprobe xhci-pci
echo 'func xhci_dbg_trace -p' >/proc/dynamic_debug/control
dmesg |grep 'page size'
[  262.117753] xhci_hcd 0000:08:00.0: Supported page size register = 0x1
[  262.117755] xhci_hcd 0000:08:00.0: Supported page size of 8K
[  262.117756] xhci_hcd 0000:08:00.0: HCD page size set to 4K

BTW, don't worry about that 8K, this value is incorrectly calculated and then completely ignored.

(In reply to Michał Pecio from comment #26)
> Honestly, I don't know what's the problem. Maybe as a last ditch effort try
> dynamic debug, if your kernel supports it:
> 
> echo 'func xhci_dbg_trace +p' >/proc/dynamic_debug/control
> rmmod xhci-pci
> modprobe xhci-pci
> echo 'func xhci_dbg_trace -p' >/proc/dynamic_debug/control
> dmesg |grep 'page size'
> [  262.117753] xhci_hcd 0000:08:00.0: Supported page size register = 0x1
> [  262.117755] xhci_hcd 0000:08:00.0: Supported page size of 8K
> [  262.117756] xhci_hcd 0000:08:00.0: HCD page size set to 4K
> 
> BTW, don't worry about that 8K, this value is incorrectly calculated and
> then completely ignored.

rolf@i7-5820K-debian-testing1:~$ sudo dmesg |grep 'page size'
[    0.189774] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.189774] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[  225.577885] xhci_hcd 0000:00:14.0: Supported page size register = 0x1
[  225.577887] xhci_hcd 0000:00:14.0: Supported page size of 8K
[  225.577888] xhci_hcd 0000:00:14.0: HCD page size set to 4K
[  225.584545] xhci_hcd 0000:05:00.0: Supported page size register = 0x1
[  225.584547] xhci_hcd 0000:05:00.0: Supported page size of 8K
[  225.584548] xhci_hcd 0000:05:00.0: HCD page size set to 4K
[  225.648450] xhci_hcd 0000:06:00.0: Supported page size register = 0x1
[  225.648452] xhci_hcd 0000:06:00.0: Supported page size of 8K
[  225.648453] xhci_hcd 0000:06:00.0: HCD page size set to 4K

(In reply to Michał Pecio from comment #26)
I have a new E-Mail-account.

This should now be fixed by
c133ec0e5717 usb: xhci: Enable the TRB overfetch quirk on VIA VL805

which hit mainline today and is bound for stable.