Created attachment 300614 [details] Journal log We have an ASUS B1400CEAE laptop equipped with Intel i5-1135G7 and Sandisk Corp WD Blue SN550 NVMe SSD. The system can resume from suspend correctly with kernel 5.10. However, it resumes failed from suspend and hangs with kernel 5.15 and current 5.17, which is a regression issue. I set the persistent journal log. I read the journal log again and again. I notice there is nothing after system becomes suspended. The kernel messages after resume might write failed to NVMe. Then, I notice the message "kernel: nvme 0000:01:00.0: platform quirk: setting simple suspend". Wonder why it quirks as simple suspend. Trace the code and disable the quirk, like diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 9f4f3884fefe..017a87ae999f 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -3096,7 +3096,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) */ dev_info(&pdev->dev, "platform quirk: setting simple suspend\n"); - quirks |= NVME_QUIRK_SIMPLE_SUSPEND; + //quirks |= NVME_QUIRK_SIMPLE_SUSPEND; } /* System resumes successfully!!! NVMe is the one which produces the bug!
Here is the information of the NVMe: 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01) (prog-if 02 [NVM Express]) Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0, IOMMU group 14 Memory at 82200000 (64-bit, non-prefetchable) [size=16K] Memory at 82204000 (64-bit, non-prefetchable) [size=256] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+ Capabilities: [b0] MSI-X: Enable+ Count=17 Masked- Capabilities: [c0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [1b8] Latency Tolerance Reporting Capabilities: [300] Secondary PCI Express Capabilities: [900] L1 PM Substates Kernel driver in use: nvme
Created attachment 300615 [details] Dump ACPI To understand why it needs the simple suspend quirk, I trace acpi_storage_d3() [1] in the if condition. I notice it goes though the whole acpi_storage_d3(), which means the ACPI node has StorageD3Enable with value 1. To confrim that, I dumped the ACPI and found it: Scope (_SB.PC00) { Device (SAT0) { Name (_ADR, 0x00170000) // _ADR: Address Name (_DSD, Package (0x02) // _DSD: Device-Specific Data { ToUUID ("5025030f-842f-4ab4-a561-99a5189762d0") /* Unknown UUID */, Package (0x01) { Package (0x02) { "StorageD3Enable", One } } }) [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/acpi/device_pm.c?h=v5.17#n1381
I found three related commits: * df4f9bc4fb9c ("nvme-pci: add support for ACPI StorageD3Enable property") * e21e0243e7b0 ("nvme-pci: look for StorageD3Enable on companion ACPI device instead") * 2744d7a07335 ("ACPI: Check StorageD3Enable _DSD property in ACPI code") The original ACPI node PXSX and PEGP check is removed in e21e0243e7b0 ("nvme-pci: look for StorageD3Enable on companion ACPI device instead") for AMD platforms. I did not see the ACPI node PXSX, nor PEGP in the ACPI tables in comment #2. So, I add the check back for test: diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c index cc6c97e7dcae..d5d93d3f01f7 100644 --- a/drivers/acpi/device_pm.c +++ b/drivers/acpi/device_pm.c @@ -1381,6 +1381,8 @@ EXPORT_SYMBOL_GPL(acpi_dev_pm_attach); bool acpi_storage_d3(struct device *dev) { struct acpi_device *adev = ACPI_COMPANION(dev); + acpi_handle handle; + acpi_status status; u8 val; if (force_storage_d3()) @@ -1388,6 +1390,14 @@ bool acpi_storage_d3(struct device *dev) if (!adev) return false; + + status = acpi_get_handle(adev->handle, "PXSX", &handle); + if (ACPI_FAILURE(status)) { + status = acpi_get_handle(adev->handle, "PEGP", &handle); + if (ACPI_FAILURE(status)) + return false; + } + if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable", &val)) return false; System resumes successfully! I think Intel platforms need the check provided by df4f9bc4fb9c ("nvme-pci: add support for ACPI StorageD3Enable property").
With the test patch in comment #3, the NVMe will not be quirked as NVME_QUIRK_SIMPLE_SUSPEND. And then, system can resume from suspend correctly. However, does this mean that the NVMe on this laptop does not support simple suspend?
If I set test modes of hibernation for PM test [1]: * devices mode: System resumes correctly # echo devices > /sys/power/pm_test # echo platform > /sys/power/disk # echo mem > /sys/power/state * platform mode: System resumes failed # echo platform > /sys/power/pm_test # echo platform > /sys/power/disk # echo mem > /sys/power/state [1] https://www.kernel.org/doc/Documentation/power/basic-pm-debugging.txt
Is this production hardware? Is this production firmware? If so - can you please confirm that you have the latest firmware from the manufacturer first in case they made a mistake in the release you have. ---- Looking at your ACPI table I see that this system likely supports s2idle as it sets: Low Power S0 Idle (V5) : 1 I have a theory here the disk is getting to the deepest state with that simple suspend set but other platform problems are causing the issue. Can you please do the following: 1) Use your workaround/revert. 2) Confirm /sys/power/mem_sleep is "s2idle" 3) Run a suspend, and then observe the values of /sys/kernel/debug/pmc_core/slp_s0_residency_usec If those are non-zero you got to the deepest state and my theory is wrong. If they're 0 it confirms my theory.
(In reply to mario.limonciello from comment #6) The firmware has been updated to the latest version 307 from official website. > Looking at your ACPI table I see that this system likely supports s2idle as > it sets: > Low Power S0 Idle (V5) : 1 > > I have a theory here the disk is getting to the deepest state with that > simple suspend set but other platform problems are causing the issue. > > Can you please do the following: > 1) Use your workaround/revert. Sure! > 2) Confirm /sys/power/mem_sleep is "s2idle" It is "s2idle". $ cat /sys/power/mem_sleep [s2idle] deep > 3) Run a suspend, and then observe the values of > /sys/kernel/debug/pmc_core/slp_s0_residency_usec > > If those are non-zero you got to the deepest state and my theory is wrong. > If they're 0 it confirms my theory. After suspend & resume, it is 0 $ sudo cat /sys/kernel/debug/pmc_core/slp_s0_residency_usec 0
Thanks, so that at least provides evidence to support my hypothesis. There's a few ways that we could approach this: 1. Use S3 instead of S2idle for this system 2. Fix the other (presumably) platform problems leading to this behavior. As the system advertises S3 as well, that's a much easier thing to try. I would suggest that you leave the code in place and just change mem_sleep to "deep" and try to suspend. If that works properly, we should be able to quirk this system to prefer "deep" even though the ACPI table advertises to use s2idle. If that doesn't help, you can try to revert the quirk and try with it reverted to see if "deep" works.
Checked the S3 with and without the workaround. $ cat /sys/power/mem_sleep s2idle [deep] System can resume from suspend. I found one of my friend placed an order for the same laptop, but under shipping. Will check with him again to make sure this is not a single hardware issue.
Created attachment 300716 [details] Patch to set default to S3 Attached a patch that should do that programmatically by default. If you can confirm this works, then after your colleague confirms it's not a hardware problem for you I can submit this up to review.
(In reply to Mario Limonciello (AMD) from comment #10) > Created attachment 300716 [details] > Patch to set default to S3 > > Attached a patch that should do that programmatically by default. If you > can confirm this works, then after your colleague confirms it's not a > hardware problem for you I can submit this up to review. Jian-Hong Pan: did you ever give this a try?
(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #11) > Jian-Hong Pan: did you ever give this a try? Sorry for the late reply. My family member got COVID, so we are in quarantine. Will have the test, when I reach the laptop in the office.
(In reply to Mario Limonciello (AMD) from comment #10) > Created attachment 300716 [details] > Patch to set default to S3 > > Attached a patch that should do that programmatically by default. If you > can confirm this works, then after your colleague confirms it's not a > hardware problem for you I can submit this up to review. Just tested the patch. My laptop can suspend & resume with the patch which makes it S3, instead of s2idle. Thanks!
Submitted up here: https://lore.kernel.org/linux-acpi/20220510131136.1103-1-mario.limonciello@amd.com/T/#u
Hey folks. I might suffer from the same issue (just with different hardware). Would be highly appreciated if the experts could have a look at: https://bugzilla.kernel.org/show_bug.cgi?id=216998 Thanks, Chris.
Hi Thorsten and Keith, Thanks for the details. I'm still unsure if responding by email is better or adding to the ticket, but here it goes: I have tried for days both with complete power off of the machine and cycle-booting all kernels in succession and without exception, 6.1.x LTS and the patched 6.5.1 kernel always recognize and operate the NVME, whilst the other kernels also fail with the same error message. As this is my "production" desktop, though, during the week it's more difficult to me to perform tests with it, but I will try to do it in a more methodic way and also with 6.5.1 vanilla. As for the reason the Lexar doesn't catch the quirk default, I can't say I catch the complex logic of the driver activation, but I found out how to "fix" for my case because there are three other Lexar models in the pci.c file: NM610, NM620 and NM760 (this one with an additional quirk marked on it on the code, NVME_QUIRK_IGNORE_DEV_SUBNQN) -- so I guess whatever justifies the exception for them also justifies for my model, NM790. Might even be the case that I would need NVME_QUIRK_IGNORE_DEV_SUBNQN (not sure what it does) like in the NM760 case, but it activates correctly without it.
This issue is long fixed; please open a new one for your problem with all the details. We need to see a full dmesg, and acpidump. There is a lot of nuance that has to do with the platform and specific hardware.
Sorry, I thought I was commenting in my original request here: https://bugzilla.kernel.org/show_bug.cgi?id=217863