Dear maintener

Here is a new bug I encountered since Linux 6.1 and I'm able to reproduce it on from Linux 6.1 to Linux 6.12-rc6 (mainline)

I have a server which is using a Broadcom PLX PEX8784 pcie switch to connect storage.
On the PEX8748, U2 NVME drives are connected.

After 3 years of perfect work in production, since I upgrade to Linux 6.1 the PEX8748 totally lost the downstream NVME drives connected to it child PCIE port.

Topology : 

PCIE root port <---- PEX8748 <---- UP to 8 NVME drives

Note that PCIE ASPM is totally disable in the UEFI BIOS since 3 years and I never had any issue before upgrading to Linux 6.1

Having a look at kernel log show that the system is puting NVME drives in D3cold state and is never able to recover from this state (even when trying to reset PCIE bus and to manualy rebind pcie and nvme drivers).

The recommanded kernel option are not efficient : "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"

And since Linux 6.1, UEFI BIOS ASPM options are also ignored and the system is doing PM on PCIE where it mustn't
And the only way to recover storage is to reboot the machine 

Kernel logs : 

kernel: nvme 0000:08:00.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: nvme 0000:0a:00.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: nvme 0000:0b:00.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: nvme 0000:09:00.0: Unable to change power state from D3cold to D0, device inaccessible

and it has the effect of breaking totally the system as you can see, multi device had lost its 4 nvme drives "[____]" on a RAID5 array :

grep -A1 md127 /proc/mdstat
 md127 : active raid5 nvme1n1p1[1](F) nvme3n1p1[4](F) nvme2n1p1[2](F) nvme0n1p1[0](F)
      12830758912 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/0] [____]


Having a look at the sysfs, I seen that d3cold_allowed is set to 1 for NVME drives and for the PEX8748 downstream port !
It seems that Windows server keep NVME in D0 state during ASPM or suspend, why not Linux when PCIE ASPM is disable ?.


Other output I capture while reproducing on a DELL laptop (with ASPM enable) and on the server (with ASPM disable): 

# NVME drive never come back to D0 state (never wakup back)
#error: 
 nvme nvme3: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
 nvme nvme3: Does your device have a faulty power saving mode enabled?
 nvme nvme3: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
 nvme 0000:0b:00.0: Unable to change power state from D3cold to D0, device inaccessible
 nvme nvme3: Disabling device after reset failure: -19

# After this failiure, it becomes impossible to wakup NVME drives without rebooting

# Exemple (NOT FROM THIS SERVER - ON LAPTOP with PCIE ASPM enable ) : 
  echo "0000:08:00.0" > /sys/bus/pci/drivers/nvme/unbind
  echo "0000:02:00.0" > /sys/bus/pci/drivers/pcieport/unbind
  setpci -s 08:00.0 CAP_PM+4.b=03 # D3hot
  acpidbg -b "execute \_SB.PC00.PEG0.PXP._OFF"
  acpidbg -b "execute \_SB.PC00.PEG0.PXP._ON"
  #debian cmd# acpiexec -b "execute \_SB.PC00.PEG0.PXP._OFF"
  #debian cmd# acpiexec -b "execute \_SB.PC00.PEG0.PXP._ON"
  setpci -s 08:00.0 CAP_PM+4.b=0 # D0
  echo "0000:02:00.0" > /sys/bus/pci/drivers/pcieport/bind
  echo "0000:08:00.0" > /sys/bus/pci/drivers/nvme/bind
  
# NVME never wakeup  here: 
 nvme 0000:08:00.0: not ready 65535ms after bus reset; giving up
 nvme 0000:08:00.0: Unable to change power state from D3cold to D0, device inaccessible
 nvme 0000:08:00.0: timed out waiting for pending transaction; performing function level reset anyway
 nvme 0000:08:00.0: not ready 1023ms after FLR; waiting
 nvme 0000:08:00.0: not ready 2047ms after FLR; waiting
 ...
 nvme 0000:08:00.0: not ready 65535ms after FLR; giving up 
 ...
 nvme 0000:08:00.0: not ready 32767ms after bus reset; waiting
 nvme 0000:08:00.0: not ready 65535ms after bus reset; giving up






While waiting for a fix in kernel, I did write the following dirty workarround to save the production (not acceptable to have a total lost of storage every morning on a production server) :

Here is my workarround :
1) identify PCIE bus and PCIE downstream port (if a PCIE switch like a PLX card is connected to the PCIE root port)  and identify the PCIE BUS address of the bind NVME drive 
2) second, write a script which disable D3cold capability for PCIE downstream port (PLX card child port, if any)
3) extend this script to disable D3cold capability on NVME drives attached to PCIE child port
4) execute script at reboot 





##############################################
#!/bin/bash
#
#  
# 20241104
# Workarround power management issue 
# NVME drive goes to d3cold and never go back to d0
#
sleep 60
echo "------------------------------------------------------"
echo "DISABLE D3COLD ON NVME U2"
echo "------------------------------------------------------"


_PLX_BUS='/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0'

_NVME7_PORT='0000:03:08.0' # UNUSED - FREE PORT
_NVME6_PORT='0000:03:09.0' # UNUSED - FREE PORT
_NVME4_PORT='0000:03:0a.0' # UNUSED - FREE PORT
_NVME5_PORT='0000:03:0b.0' # UNUSED - FREE PORT

_NVME0_PORT='0000:03:10.0'
_NVME1_PORT='0000:03:11.0'
_NVME2_PORT='0000:03:12.0'
_NVME3_PORT='0000:03:13.0'


_NVME7_DEV='0000:04:00.0' # NOT BIND
_NVME6_DEV='0000:05:00.0' # NOT BIND
_NVME4_DEV='0000:06:00.0' # NOT BIND 
_NVME5_DEV='0000:07:00.0' # NOT BIND

_NVME0_DEV='0000:08:00.0'
_NVME1_DEV='0000:09:00.0'
_NVME2_DEV='0000:0a:00.0'
_NVME3_DEV='0000:0b:00.0'

# First disable D3COLD on NVME PLUGGED PCIE port
# For the moment, we only consider the 4 plugged NVME port 
# num_port => 4

	echo "DISABLE D3COLD ON PLX CHILD PCIE PORT:"
	echo 0 >"$_PLX_BUS/$_NVME0_PORT/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME1_PORT/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME2_PORT/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME3_PORT/d3cold_allowed"

# Control

	echo "$_PLX_BUS/$_NVME0_PORT/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME0_PORT/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME1_PORT/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME1_PORT/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME2_PORT/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME2_PORT/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME3_PORT/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME3_PORT/d3cold_allowed"

# Second disable D3COLD on CONNECTED NVME drives
# For the moment, we only consider the 4 plugged NVME port 
# num_dev => 4

	echo "DISABLE D3COLD ON NVME DRIVE ATTACHED TO PLX CHILD PCIE PORT:"
	echo 0 >"$_PLX_BUS/$_NVME0_PORT/$_NVME0_DEV/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME1_PORT/$_NVME1_DEV/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME2_PORT/$_NVME2_DEV/d3cold_allowed"
	echo 0 >"$_PLX_BUS/$_NVME3_PORT/$_NVME3_DEV/d3cold_allowed"

# Control

	echo "$_PLX_BUS/$_NVME0_PORT/$_NVME0_DEV/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME0_PORT/$_NVME0_DEV/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME1_PORT/$_NVME1_DEV/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME1_PORT/$_NVME1_DEV/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME2_PORT/$_NVME2_DEV/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME2_PORT/$_NVME2_DEV/d3cold_allowed"
	echo "$_PLX_BUS/$_NVME3_PORT/$_NVME3_DEV/d3cold_allowed:"
	cat "$_PLX_BUS/$_NVME3_PORT/$_NVME3_DEV/d3cold_allowed"


#
echo -e "\n---------------------------------------------"
echo "SUCESSFULLY DISABLED D3COLD ON NVME U2"
echo "---------------------------------------------"

#######################################################


Hope a fix could be written, everything was working fine since 3 years with Linux 5.X and 6.0.X and the first time the isssue happend was just apter upgrading to Linux 6.1.20 and I reproduce it on Linux 6.10.11 and also in mainline (6.12rc6)

Thanks for help
Kind regards
Nicolas BARANGER