Hi, I see _tons_ of the following messages in kernel log: [71927.729772] nvme 0000:01:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8 GT/s x4 link) [71927.770321] nvme 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8 GT/s x4 link) [71941.013453] nvme 0000:01:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8 GT/s x4 link) [71941.054245] nvme 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8 GT/s x4 link) the messages always appear in pairs, first 16 Gbps then 8Gbps. The corresponding device is an NVMe controller: 01:00.0 Non-Volatile memory controller: Lenovo Device 0003 The NVMe performance is also reduced (500-800 MB/s when the laptop is plugged in). I found an old benchmark (cca August) that showed closer to 1.6/2 GB/s on the same machine. Is there any wa to find ou why the link speed has been reduced? (hw issue, power saving setting)
Thanks a lot for the report! Can you try turning off CONFIG_PCIE_BW? That should turn off the log messages. I'm interested to know whether that also affects the NVMe performance. Even if it helps, turning off CONFIG_PCIE_BW is not a very good fix. I don't think it's reasonable for users or distros to figure out whether that option is safe. See 2078e1e7f7e0 ("PCI/LINK: Add Kconfig option (default off)"), https://git.kernel.org/linus/2078e1e7f7e0
Hi, thanks for a prompt response. changing CONFIG_PCIE_BW did remove the messages. hdparm run bufferred reads showed about the same performance before and after. Since it could get ~800MB/s (and using gnome disk tool I could get a an average rate of ~900 MB/s.) I think it's mostly limited by the available PCIE bw. A log message would be nice if it didn't happen so often. which brings me to the question what controls the link speed? bios? driver? is it dependent on aspm setting? During repeated runs of the disk read benchmark I could occasionally bw go up to 1800MB/s. checking lspci confirmed that LnkSta: Speed 2.5GT/s (downgraded), Width x4 (ok) changed into: LnkSta: Speed 5GT/s (downgraded), Width x4 (ok) thank you
The performance is less clear when you're using buffered reads. An O_DIRECT test would isolate how the device is doing. This could be in response to a power saving scheme on the end device. You can disable nvme autonomous power state transitions (apst) with kernel parameter "nvme_core.default_ps_max_latency_us=0" and see if that resolves your observations. If that doesn't work, you can try to disable generic pcie aspm with kernel param "pcie_aspm=off". These things can also happen from platform firmware/bios too, so maybe the above won't help.
Jan, can you rule out a hardware issue with the link? Do you see the same message flood in a different slot or different machine? It'd be ideal if you could reproduce this with a different nvme device in a different machine.
Hi, sorry for the delay. I assumed that using gnome disks benchmark tool would take care of the right diska ccess mode. I tried booting with pcie_aspm=off and it didn't change the observed behaviour. OTOH, kernel log says even without the parameter so it looks like linux doesn't have control over aspm to begin with: [ 0.727756] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it and [ 0.887906] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration I can try looking for another slot but it's a laptop machine (Lenovo E485) so I'm not sure there will be a way to swap it around. The rate of messages is highly correlated with storage IO (heavier IO -> more messages), and there were several BIOS updates since Feb 2019 when I measured up to 2.5 GB/s read throughput. Jan
(In reply to Jan Vesely from comment #5) > I tried booting with pcie_aspm=off and it didn't change the observed > behaviour. What about the nvme apst option? That one is a more device specific runtime power state control.
What's the status of this? Did it get resolved? It *seems* like this case might be related to an NVMe issue, not a PCI core issue? Or do we need an interface so a driver can say "I'm going to intentionally reduce the link speed"?
Hi, Recently I found that I have the same issue as the OP. The machine is Lenovo A485 with the latest BIOS (1.33). The NVMe drive is SAMSUNG MZFLV512HCJH-000MV, and I have Linux Mint 19.3 with 5.10.0-11.3-liquorix-amd64 kernel. I've tried both "pcie_aspm=off" and "nvme_core.default_ps_max_latency_us=0" without any luck. Here is how the messages are looking: ``` Jan 28 19:45:27 A485 kernel: nvme 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x2 link at 0000:00:01.3 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) Jan 28 19:45:32 A485 kernel: nvme 0000:03:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:01.3 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) Jan 28 19:45:32 A485 kernel: nvme 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x2 link at 0000:00:01.3 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) Jan 28 19:45:37 A485 kernel: nvme 0000:03:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:01.3 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) ``` Any help will be much appreciated.