Created attachment 307967 [details] 6.14.2-300.fc42.x86_64 journalctl -k output Upgrading to 6.14.2 results in any USB device plugged into a USB hub to not work correctly. Downgrading to 6.13.7-200.fc41.x86_64 allows for all USB devices to work as expected. This issue also seems to be present in 6.15.0 as well. Here is the downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=2353984 with more details. Another user on the RH BZ with a very similar motherboard is reporting that none of their USB devices are working when upgrading to 6.14.2 I have attached the journalctl -k output for 6.13.7-200.fc41.x86_64 and one thing that seems notable is: Apr 14 20:06:49 kernel: ahci 0000:26:00.0: version 3.0 Apr 14 20:06:49 kernel: ahci 0000:26:00.0: probe with driver ahci failed with error -12 Apr 14 20:06:49 kernel: ahci 0000:27:00.0: probe with driver ahci failed with error -12 Apr 14 20:06:49 kernel: xhci_hcd 0000:23:00.0: init 0000:23:00.0 fail, -16 Apr 14 20:06:49 kernel: xhci_hcd 0000:23:00.0: probe with driver xhci_hcd failed with error -16 Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.1: init 0000:28:00.1 fail, -16 Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.1: probe with driver xhci_hcd failed with error -16 Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.3: init 0000:28:00.3 fail, -16 Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.3: probe with driver xhci_hcd failed with error -16 and in the 6.13.7-200.fc41.x86_64 output, this is not present. I have attached the 6.13.7-200.fc41.x86_64 journalctl -k output as well and the lspci -v output is below. # lspci -v|rg -A 15 USB 06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device 8815 Flags: bus master, fast devsel, latency 0, IRQ 94, IOMMU group 85 Memory at f3600000 (64-bit, non-prefetchable) [size=1M] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [64] Express Endpoint, IntMsgNum 0 Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=8 Masked- Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [2a0] Access Control Services Capabilities: [370] Transaction Processing Hints Kernel driver in use: xhci_hcd -- 23:00.0 USB controller: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device 87c1 Flags: bus master, fast devsel, latency 0, IRQ 65, IOMMU group 51 Memory at f0600000 (64-bit, non-prefetchable) [size=32K] Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [68] MSI-X: Enable+ Count=8 Masked- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, IntMsgNum 0 Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 0201 Capabilities: [100] Advanced Error Reporting Capabilities: [200] Secondary PCI Express Capabilities: [300] Latency Tolerance Reporting Capabilities: [400] L1 PM Substates Kernel driver in use: xhci_hcd -- 28:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device 8815 Flags: bus master, fast devsel, latency 0, IRQ 75, IOMMU group 50 Memory at f0100000 (64-bit, non-prefetchable) [size=1M] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [64] Express Endpoint, IntMsgNum 0 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: xhci_hcd 28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI]) Subsystem: Advanced Micro Devices, Inc. [AMD] Device 148c Flags: bus master, fast devsel, latency 0, IRQ 76, IOMMU group 50 Memory at f0000000 (64-bit, non-prefetchable) [size=1M] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [64] Express Endpoint, IntMsgNum 0 Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=8 Masked- Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: xhci_hcd -- 2c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device 8815 Flags: bus master, fast devsel, latency 0, IRQ 85, IOMMU group 62 Memory at f0700000 (64-bit, non-prefetchable) [size=1M] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [64] Express Endpoint, IntMsgNum 0 Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=8 Masked- Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [2a0] Access Control Services Capabilities: [370] Transaction Processing Hints Kernel driver in use: xhci_hcd
Created attachment 307968 [details] 6.13.7-200.fc41.x86_64 journalctl -k output
Would be great if you could bisect: https://docs.kernel.org/admin-guide/bug-bisect.html
What if you disconnect the hub and connect an ordinary device in its place? This log looks like several USB ports on your computer should be completely dead. And some SATA ports too. This looks like devm_request_mem_region() is failing in usb_hcd_pci_probe(), and maybe pcim_iomap() in ahci (SATA). So something to do with PCI, ACPI or other lower level stuff. You may try your luck reassigning this bug to drivers/PCI, they seem to be responsive to bugzilla reports.
(In reply to Michał Pecio from comment #3) > What if you disconnect the hub and connect an ordinary device in its place? > This log looks like several USB ports on your computer should be completely > dead. And some SATA ports too. I just swapped my mouse into the USB A 3.0 port that my hub was in and it is dead too. > This looks like devm_request_mem_region() is failing in usb_hcd_pci_probe(), > and maybe pcim_iomap() in ahci (SATA). So something to do with PCI, ACPI or > other lower level stuff. > > You may try your luck reassigning this bug to drivers/PCI, they seem to be > responsive to bugzilla reports. OK sounds good. I will reassign this to drivers/PCI to see if those folks have any ideas what is going on here.
Created attachment 307975 [details] Good boot with 6.13 kernel
Created attachment 307976 [details] Bad boot with 6.14 kernel
I appear to be having the same problem, and I've attached journal logs from a good boot and a bad boot. In my case I have a mouse and keyboard directly plugged into the motherboard. The keyboard was not working once I upgraded to kernel 6.14 but the mouse was working. I stepped back to kernel 6.13, and both mouse and keyboard were working. I tried switching which connectors the mouse and keyboard were plugged into, and at that point the mouse wasn't working but the keyboard was working. So with kernel 6.14 several of the motherboard jacks were not functional. I looked at the pci information, and the usb devices are: usb1/2 0000:23:00.0 ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller usb3/4 0000:28:00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller usb5/6 0000:28:00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller usb7/8 0000:2d:00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller usb9/10 0000:05:00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller Apparently, with 6.14 one of the Matisse controllers was recognized, but the other was not. And neither of the Starship controllers were recognized.
I am now able to build a working 6.14 kernel. The only thing I had to change was to turn off CONFIG_PCI_REALLOC_ENABLE_AUTO. With CONFIG_PCI_REALLOC_ENABLE_AUTO off, all my USB controllers work. With CONFIG_PCI_REALLOC_ENABLE_AUTO=y, some of my USB controllers don't initialize.
Setting pci=realloc=off in the kernel cmdline works as well.
The question now is whether this is a kernel bug or a motherboard bug. If it is a mobo bug then I guess a quirk would be appropriate.
Does also 6.13 fail if pci=realloc is on the kernel command line? It's not clear to me if PCI resource reallocation got enabled along with the kernel upgrade, that is, if Fedora packaging e.g. decided to enable CONFIG_PCI_REALLOC_ENABLE_AUTO in kernel's .config which is orthogonal to kernel version changes. Please provide /proc/iomem from both working and non-working configuration. Preferrably with the otherwise identical kernel version, if you can't get it from the non-working configuration at all due to lack of keyboard, I'll manage with the log from working only (things can be inferred from dmesg but it's very very tedious process). Could you also clarify what you meant with this: "This issue also seems to be present in 6.15.0 as well."? Kernel version 6.15 is not released yet, 6.15 cycle is only at -rc2 at this point. I'll try to look more into this next week.
Created attachment 307986 [details] 6.13.10 with realloc=on, no keyboard
Created attachment 307987 [details] 6.13.10 with realloc=off, good keyboard
Created attachment 307988 [details] 6.14.2 with realloc=on, no keyboard
Created attachment 307989 [details] 6.14.2 with realloc=off, good keyboard
I ran the tests you requested. I attached /proc/iomem for 6.13.10 both with realloc=off and realloc=on. I repeated the tests with 6.14.2. For both kernel versions, realloc=on results in a non-functional USB keyboard. For both kernels, realloc=off works properly. I had been running Fedora 41 with their 6.13 kernel. It was compiled with realloc=off. I then upgraded to Fedora 42 with their 6.14 kernel. It was compiled with realloc=on. Therefore, this is not a new problem with 6.14; it is just that the kernel option was changed in Fedora 42. I cannot comment about 6.15 but I think Joe was the one who tried it. Perhaps he can comment on that. However, since both 6.13 and 6.14 don't work if realloc=on, it probably doesn't matter. Please let me know if there is any other data you need.
I just realized that I need to read /proc/iomem as root. I'll recreate the attachments - please give me a few minutes.
Created attachment 307990 [details] 6.13.10 with realloc=on, no keyboard
Created attachment 307991 [details] 6.13.10 with realloc=off, good keyboard
Created attachment 307992 [details] 6.14.2 with realloc=on, no keyboard
Created attachment 307993 [details] 6.14.2 with realloc=off, good keyboard
I updated the attachments.