Bug 213047

Summary: VMD created PCI bridge lacks ACPI companion device - no d3cold support for NVMe
Product: Power Management Reporter: David Box (david.e.box)
Component: OtherAssignee: David Box (david.e.box)
Status: ASSIGNED ---    
Severity: normal CC: bijidroid, cx, godmar, jonathan.derrick, morgoth6, rjw, rui.zhang, wendy.wang
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 5.12 Subsystem:
Regression: No Bisected commit-id:
Attachments: DSDT
SSDT containing VMD power resource objects
PCI: VMD: Set ACPI companion for the VMD host bridge

Description David Box 2021-05-12 19:27:31 UTC
When operating under VMD, the root port for the NVMe doesn't have an ACPI companion device and therefore lacks support do D3 cold.

Under AHCI mode we see the following:

/sys/devices/pci0000:00/0000:00:1d.0/firmware_node/adr:0x001d0000
/sys/devices/pci0000:00/0000:00:1d.0/firmware_node/path:\_SB_.PC00.RP13
/sys/devices/pci0000:00/0000:00:1d.0/0000:01:00.0/firmware_node/adr:0x00000000
/sys/devices/pci0000:00/0000:00:1d.0/0000:01:00.0/firmware_node/path:\_SB_.PC00.RP13.PXSX

Under VMD mode however, we only see this:

/sys/devices/pci0000:00/0000:00:0e.0/firmware_node/adr:0x000e0000
/sys/devices/pci0000:00/0000:00:0e.0/firmware_node/path:\_SB_.PC00.VMD0


The VMD driver creates a new host bridge starting on bus 10000:
10000:e0:17.0 SATA controller: Intel Corporation Device 7ae2 (rev 10)
10000:e0:1d.0 System peripheral: Intel Corporation Device 09ab
10000:e0:1d.4 PCI bridge: Intel Corporation Device 7ab4 (rev 10)
10000:e1:00.0 Non-Volatile memory controller: SK hynix Device 1627


But none of these devices have ACPI objects associated with the, despite their appearance in the acpidump.
Comment 1 David Box 2021-05-12 19:28:44 UTC
Created attachment 296731 [details]
DSDT
Comment 2 David Box 2021-05-12 19:29:33 UTC
Created attachment 296733 [details]
SSDT containing VMD power resource objects
Comment 3 Rafael J. Wysocki 2021-05-14 13:55:50 UTC
Created attachment 296747 [details]
PCI: VMD: Set ACPI companion for the VMD host bridge

If I'm not mistaken, this may be as simple as the attached patch.

The _PS0 and _PS3 objects under the ACPI device in question are a bit of a concern, but AFAICS the PCI layer doesn't do any PM of the host bridge devices, so they should not be evaluated in that context.

Please test and let me know how it goes.
Comment 4 David Box 2021-05-15 20:17:21 UTC
The patch did add a companion device to the VMD host bridge but this was not enough to find companions for the devices on the bridge. I was able to determine that this is because the _ADRs for these devices don't match the expected format. The VMD folks did say that they use the following convention:

For PCIE: 8xyzFFFF, where xyz is the BDF 0Bbbbb'ddddd'fff. 
For SATA: 0xyxFFFt, where t is the port and xyz is the BDF as above.

But this doesn't match the ACPI spec. I enabled the ACPI debug messages (ADBG) to print the _ADRs for the devices remapped under VMD. They are in the "VMD _ADR" column below.

                | Bus Location  | ACPI _ADR  | VMD _ADR
PCIe Root Port  | 10000:e0:6.0  | 0x60000    | 0x8030FFFF 
SATA Controller	| 10000:e0:17.0 | 0x17000    | 0XB8FFF0

Because of this mismatch, the devices are not found by pci_acpi_find_companion(). The ACPI _ADR does convert the the VMD _ADR using the VMD convention above. When I hacked the function to do this conversion for devices on domain 0x10000, they were able to find their ACPI companions devices. And the root port was able to go to D3 cold on suspend. But this is of course not standard. We will need to get documentation for this. The VMD driver may need to implement its own acpi_bus_type in order to do the expected _ADR search.
Comment 5 Adam Štrauch 2021-07-18 20:49:44 UTC
Hi,

would be possible to post here the mentioned hack? Unfortunately my notebook is affected by this bug and HP does't allow me to disable VMD from BIOS setup so that hack is probably the only option I could use until it's properly fixed.
Comment 6 David Box 2021-07-19 20:11:26 UTC
(In reply to Adam Štrauch from comment #5)
> Hi,
> 
> would be possible to post here the mentioned hack? Unfortunately my notebook
> is affected by this bug and HP does't allow me to disable VMD from BIOS
> setup so that hack is probably the only option I could use until it's
> properly fixed.

This particular VMD issue should not affect notebooks. Can you open a new bug report for your HP system? If you think it's VMD related assign it to Power Management
Comment 7 Adam Štrauch 2021-07-19 20:49:00 UTC
I already did and I was pointed into this bug report there :-)

https://bugzilla.kernel.org/show_bug.cgi?id=213717
Comment 8 Godmar Back 2023-01-08 15:15:56 UTC
I'd like to leave a note that this issue persists in 6.1.1 and affects, for instance, a Dell Latitude 9520 with BIOS 0.17.1.  Work-around is described in comments 24 and 32

https://bugzilla.kernel.org/show_bug.cgi?id=211879
Comment 9 Khad 2023-02-03 10:37:55 UTC
I also can't reach s0ix, laptop is Vivobook_ASUSLaptop X3400PH_K3400PH. 

Workaround is by running:
echo 8 >  /sys/kernel/debug/pmc_core/ltr_ignore

But this is not allowed when Secure Boot active.

While disabling VMD, there is warning: this will make system malfunction, so I didn't disable it. my system is dualboot with windows.