Bug 205587 - AMD Ryzen5/7 XHCI controllers fail to runtime resume
Summary: AMD Ryzen5/7 XHCI controllers fail to runtime resume
Status: RESOLVED CODE_FIX
Alias: None
Product: Power Management
Classification: Unclassified
Component: Run-Time-PM (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Rafael J. Wysocki
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-20 09:15 UTC by Daniel Drake
Modified: 2019-12-16 09:10 UTC (History)
0 users

See Also:
Kernel Version: 5.3
Tree: Mainline
Regression: No


Attachments
UX434DA lspci output (43.30 KB, text/plain)
2019-11-20 09:17 UTC, Daniel Drake
Details
UX434DA acpidump (704.14 KB, text/plain)
2019-11-20 09:20 UTC, Daniel Drake
Details
UX434DA dmesg (69.50 KB, text/plain)
2019-11-27 03:38 UTC, Daniel Drake
Details
X512DK dmesg (73.74 KB, text/plain)
2019-11-27 05:30 UTC, Daniel Drake
Details
X512DK lspci (44.87 KB, text/plain)
2019-11-27 05:31 UTC, Daniel Drake
Details

Description Daniel Drake 2019-11-20 09:15:44 UTC
On Asus UX434DA (AMD Ryzen7 3700U) and Asus X512DK (AMD Ryzen5 3500U), the XHCI controller fails to resume from runtime suspend or s2idle, and USB becomes unusable from that point.

These are new platforms, here is no known working previous version.

xhci_hcd 0000:03:00.4: Refused to change power state, currently in D3
xhci_hcd 0000:03:00.4: enabling device (0000 -> 0002)
xhci_hcd 0000:03:00.4: WARN: xHC restore state timeout
xhci_hcd 0000:03:00.4: PCI post-resume error -110!
xhci_hcd 0000:03:00.4: HC died; cleaning up

Can be reproduced with s2idle or even just runtime pm:
# echo on > /sys/bus/pci/devices/0000\:03\:00.3/power/control
# echo auto > /sys/bus/pci/devices/0000\:03\:00.3/power/control
# echo 1 > /sys/bus/usb/devices/1-4/remove
# cat /sys/bus/pci/devices/0000\:03\:00.3/power/runtime_status
suspended
# echo on > /sys/bus/pci/devices/0000\:03\:00.3/power/control
<fail>

This has been discussed in some detail in email threads:
Ryzen7 3700U xhci fails on resume from sleep
[PATCH] PCI: also apply D3 delay when leaving D3cold
[PATCH] PCI: PM: Fix pci_power_up()
[PATCH] PCI: increase D3 delay for AMD Ryzen5/7 XHCI controllers
Comment 1 Daniel Drake 2019-11-20 09:17:58 UTC
Created attachment 285981 [details]
UX434DA lspci output
Comment 2 Daniel Drake 2019-11-20 09:20:10 UTC
Created attachment 285985 [details]
UX434DA acpidump
Comment 3 Daniel Drake 2019-11-22 02:50:31 UTC
The workaround being explored is increasing the pci_dev_d3_sleep() delay from 10ms to 20ms. That completely avoids the problem.

Trying to understand this better and look for alternatives to quirking the delay on these XHCI controllers, testing with pci.git pci/pm HEAD 395f121e6199 which includes

  bae26849372b ("PCI/PM: Move pci_dev_wait() definition earlier")
  395f121e6199 ("PCI/PM: Wait for device to become ready after power-on")

When runtime suspending the device:
1. pci_raw_set_power_state() transitions from D0 to D3hot
2. acpi_device_set_power() calls _PS3 to transition to D3hot, then calls acpi_power_transition(D3cold) to remove device power

When runtime resuming the device:
1. acpi_device_set_power() calls acpi_power_transition(D0) to power on the device, then calls _PS0 to transition to D0.
2. pci_update_current_state() is called, reads pmcsr and sets dev->current_state=D3hot.
3. pci_dev_wait() is called but as COMMAND=0x100403 it does nothing
4. pci_raw_set_power_state(D0) is called. pmcsr is 0x103. pci_dev_d3_sleep() delays 10ms. now pmcsr has value 0x3. This causes the log message "refused to change power state" and USB is unusable from this point.

The 2 ways to detect that the failed resume condition has been met are:
1. PCI_COMMAND reg. This starts with value 0x403 right before pmcsr is written to transition to D0. After the 10ms delay, it still has value 0x403. This is an indication that the failure condition is going to be met. After the extra 10ms elay, PCI_COMMAND has value 0, which indicates that things are OK to continue.

2. PCI_PM_CTRL reg. This starts with value 0x103. Then 0 is written and pci_dev_d3_sleep() delays 10ms. At this point it has value 0x3, which is an indication that the failure condition is going to be met, and Linux already logs "refused to change power state" here. After an additional 10ms delay, it has value 0.

So, as an alternative to the device quirk, we could examine either of those registers to detect the condition, and if detected, either delay another 10ms or completely retry the pmcsr D0 transition again (which will also have the required effect of delaying another 10ms).

There has been no observed evidence that this condition can be detected as CRS. At all existing points in the code where pci_dev_wait() is called, PCI_COMMAND does not have the required ~0 value that would cause a delay. Same for PCI_VENDOR_ID. I also checked reading PCI_VENDOR_ID before the D0 transition, and during the D0 transition, and after, it is always value 0x1022 i.e. does not provide an indication that this failure is met.
Comment 4 Daniel Drake 2019-11-27 03:38:42 UTC
Created attachment 286077 [details]
UX434DA dmesg
Comment 5 Daniel Drake 2019-11-27 05:30:51 UTC
Created attachment 286081 [details]
X512DK dmesg
Comment 6 Daniel Drake 2019-11-27 05:31:10 UTC
Created attachment 286083 [details]
X512DK lspci
Comment 7 Daniel Drake 2019-11-27 05:33:28 UTC
Checking on the D3cold vs D3hot aspect of things, with ACPI dbug messages from device_pm.c  I can see that Linux is turning off the relevant power domains when going into D3cold.

However, I'm rather convinced that this is not actually fully cutting power to the PCI device. Experimenting, I add a 200ms delay and then a pmscr register read to the end of pci_set_power_state() , after pci_platform_power_transition() has been called. If the power was truly cut and we're in D3cold then I would
expect this to fail. However the register read succeeds and returns
the same value 0x103.

So on these platforms the device really suspends into D3hot and then we have trouble doing the D3hot to D0 transition, where an extra delay is needed.
Comment 8 Daniel Drake 2019-12-16 09:10:07 UTC
Fixed by
 PCI: increase D3 delay for AMD Ryzen5/7 XHCI controllers

Note You need to log in before you can comment on or make changes to this bug.