Bug 201527

Created attachment 279159 [details]
lspci with ACPI enabled

Created attachment 279161 [details]
dmesg log with ACPI disabled

Created attachment 279163 [details]
lspci with ACPI disabled

Created attachment 279165 [details]
dmesg log with ACPI enabled *default bios settings for TB*

Hello,

I realized that I had been tweaking many of the Thunderbolt BIOS settings in order to try to get things to work.  Specifically Thunderbolt Adapter Boot Support and Thunderbolt Adapter Pre-Boot Module Support.

I've reset my BIOS to fatory defaults, rebooted, plugged in a TB device.  Now it looks like the PCI resource issues are present for *all* the TB PCI bridges.

Here are the lines from dmesg that look relevant to me:

[   84.162294] pci_bus 0000:05: Allocating resources
[   84.162334] pci 0000:05:01.0: bridge window [io  0x1000-0x0fff] to [bus 07-39] add_size 1000
[   84.162344] pci 0000:05:02.0: bridge window [io  0x1000-0x0fff] to [bus 3a] add_size 1000
[   84.162346] pci 0000:05:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000 add_align 100000
[   84.162356] pci 0000:05:04.0: bridge window [io  0x1000-0x0fff] to [bus 3b-6e] add_size 1000
[   84.162366] pci 0000:04:00.0: bridge window [io  0x1000-0x0fff] to [bus 05-6e] add_size 3000
[   84.162369] pci 0000:04:00.0: BAR 13: no space for [io  size 0x3000]
[   84.162369] pci 0000:04:00.0: BAR 13: failed to assign [io  size 0x3000]
[   84.162371] pci 0000:04:00.0: BAR 13: no space for [io  size 0x3000]
[   84.162371] pci 0000:04:00.0: BAR 13: failed to assign [io  size 0x3000]
[   84.162375] pci 0000:05:02.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[   84.162376] pci 0000:05:02.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[   84.162377] pci 0000:05:01.0: BAR 13: no space for [io  size 0x1000]
[   84.162377] pci 0000:05:01.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162378] pci 0000:05:02.0: BAR 13: no space for [io  size 0x1000]
[   84.162379] pci 0000:05:02.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162380] pci 0000:05:04.0: BAR 13: no space for [io  size 0x1000]
[   84.162381] pci 0000:05:04.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162382] pci 0000:05:04.0: BAR 13: no space for [io  size 0x1000]
[   84.162383] pci 0000:05:04.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162384] pci 0000:05:02.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[   84.162385] pci 0000:05:02.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[   84.162386] pci 0000:05:02.0: BAR 13: no space for [io  size 0x1000]
[   84.162387] pci 0000:05:02.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162388] pci 0000:05:01.0: BAR 13: no space for [io  size 0x1000]
[   84.162388] pci 0000:05:01.0: BAR 13: failed to assign [io  size 0x1000]
[   84.162390] pci 0000:05:00.0: PCI bridge to [bus 06]
[   84.162396] pci 0000:05:00.0:   bridge window [mem 0xea000000-0xea0fffff]
[   84.162406] pci 0000:05:01.0: PCI bridge to [bus 07-39]
[   84.162411] pci 0000:05:01.0:   bridge window [mem 0xbc000000-0xd3efffff]
[   84.162432] pci 0000:05:01.0:   bridge window [mem 0x2fb0000000-0x2fcfffffff 64bit pref]
[   84.162441] pci 0000:05:02.0: PCI bridge to [bus 3a]
[   84.162448] pci 0000:05:02.0:   bridge window [mem 0xd3f00000-0xd3ffffff]
[   84.162457] pci 0000:05:04.0: PCI bridge to [bus 3b-6e]
[   84.162463] pci 0000:05:04.0:   bridge window [mem 0xd4000000-0xe9ffffff]
[   84.162467] pci 0000:05:04.0:   bridge window [mem 0x2fd0000000-0x2ff9ffffff 64bit pref]
[   84.162473] pci 0000:04:00.0: PCI bridge to [bus 05-6e]
[   84.162478] pci 0000:04:00.0:   bridge window [mem 0xbc000000-0xea0fffff]
[   84.162482] pci 0000:04:00.0:   bridge window [mem 0x2fb0000000-0x2ff9ffffff 64bit pref]

Just wanted to confirm that the problem is more than just device 0000:05:02.0 like I had originally thought.

Thanks!
Rob

Created attachment 279167 [details]
dmesg log with ACPI enabled *default bios settings for TB* CLEAN

A clean version of the previous dmesg log with ACPI enabled and default BIOS settings.  No eGPU was connected the system was simply booted.

One more detail that I didn't include.  I was running the latest TB controller firmware for the XPS 9575, which is NVM 36 when I created this bug report.  I updated from NVM 30 to 36 using fwupdmgr.

XPS 9575 Thunderbolt Controller
  DeviceId:             069ac71f347e92d158f2c211cca10d52a19e2d41
  Guid:                 8926f505-8219-5d6c-969a-e927534113fb
  Summary:              Unmatched performance for high-speed I/O
  Plugin:               thunderbolt
  Flags:                internal|updatable|supported|registered
  Vendor:               Dell
  VendorId:             TBT:0x00D4
  Version:              36.00
  Icon:                 computer
  Created:              2018-10-24
  Modified:             2018-10-24
  UpdateState:          success

I'm running the same system, with an MSI Armor MK2 8G OC in an Mantiz MZ-02 eGPU box.  Everything but the external PCI-E lane comes up (SSD, USB ports).  The card will initialize and be usable, but terrible performance, with pci=noacpi.

Hi,

The 0x00200000 failure is actually not fatal. It is just Linux PCI core that tries to add some additional resources but failing to add them is just fine. The requested resources are still allocated.

I don't see in your dmesg that you are actually connecting any Thunderbolt device. Can you connect the device that fails and then post full dmesg and output of 'sudo lspci -vv' command?

Thanks!

Hi Mika,

Thanks for the response.  I've actually had to return my original (AMD RX 580) because I was unable to get it working, but I did keep the TB enclosure and have access to another eGPU (Nvidia GTX 1060) that I can test with.  I'll make sure to report back with my findings.

One important thing worth mentioning is that I had originally opened up a bug report for DRI/amdgpu - located here: https://bugs.freedesktop.org/show_bug.cgi?id=108521

At the time the AMD developers thought (and it seems reasonable) that root cause of the issue was related to the PCI resource issues, and more specifically an ACPI BIOS bug - which is why I've opened up this bug report.

I still have lots of information from that original bug report that I'm attaching here, the main difference is that I'm now running kernel 4.19.4 while those logs, etc. were from kernel 4.19.0.

Thanks!
Rob

Created attachment 279745 [details]
dmesg log eGPU AMD RX 580 attached ACPI enabled

The attempted initialization of the RX 580 (as an eGPU over TB) happens at timestamp [   11.118853]

The initialization fails with these messages:

[   16.124778] [drm:atom_op_jump [amdgpu]] *ERROR* atombios stuck in loop for more than 5secs aborting
[   16.124805] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing E650 (len 187, WS 0, PS 4) @ 0xE6FA
[   16.124828] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing C53A (len 193, WS 4, PS 4) @ 0xC569
[   16.124851] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing C410 (len 114, WS 0, PS 8) @ 0xC47C
[   16.124853] amdgpu 0000:09:00.0: gpu post error!
[   16.124854] amdgpu 0000:09:00.0: Fatal error during GPU init

The AMD developers believe the issue might be caused by PCI BAR resource issues - specifically some of the PCI to TB bridges not getting the necessary resources.

Created attachment 279747 [details]
lspci eGPU AMD RX 580 attached ACPI enabled

lspci -t -nn -v output when the eGPU AMD RX 580 is connected.

Since you don't have the AMD thing anymore I don't think it makes much sense to investigate that. Can you attach dmesg with the NVIDIA one and please include output of 'sudo lspci -vv' so that I can check the PCIe configuration. Assuming it does not work with the NVIDIA card.

Apologies for the late reply.  I've been out of town for the past month.

I ended up giving away the Nvidia GPU to a friend of mine over the holidays, but I will be purchasing another AMD GPU (probably the Vega 56) in the near future.

I'll make sure to update this bug report once I've tried to get that card setup as an eGPU.  I'll also make sure to conduct my testing against 4.20 rather than 4.19.

I have same problem but with Vega 56. Laptop is almost the same Precision 5530 2-in-1 (only cpu is different). I have tested it with 4.18, 4.19 and 5.0.4 and 5.0.5 with no success.

I'm sorry it's taken me so long to update this bug report.

After spending entirely too long trying to get eGPUs working with my Dell XPS 9575, I have been able to get one working correctly - but only a Nvidia GPU.

I purchased an RTX 2070, and combined with the Akitio Node 2 it works correctly (albeit with some Xorg.conf tweaks).

I'll include the dmesg from my system here as a reference (with the nvidia GPU as an eGPU), but I believe we can close this bug report, as the Dell BIOS does not appear to be an issue here - at least not with a Nvidia GPU.

I think the bug over here still stands: https://bugs.freedesktop.org/show_bug.cgi?id=108521

Created attachment 282507 [details]
dmesg log with kernel 5.0.x with nvidia eGPU

Created attachment 282509 [details]
lspci kernel 5.0.x with nividia eGPU

I have most likely the same issue. Works on 5.7 and broken on 5.8 with my nvme thunderbolt drive which I rarely disconnect so I didn't notice it until now.

Bisect is incoming.

I have a very similar resource allocation issue on a completely different platform (ASRock Creator X570, Ryzen 3950X, Razer Core X Chroma, NVidia Quadro P5000). Here are my posts on egpu.io with dmesg and other details:
https://egpu.io/forums/postid/90573/
https://egpu.io/forums/postid/90608/

The kernel command line magic I’ve been using:
pcie_ports=native pci=assign-busses,hpbussize=0x33,realloc,hpmmiosize=128M,hpmmioprefsize=16G

The number of errors in dmesg depends on hpmmiosize and hpmmioprefsize like this (based on what I tried, which is in no way exhaustive):

256M / 64G: A huge number of errors affecting multiple BARs, even before the eGPU connection: https://pastebin.com/kcdHADw6 The eGPU works though.

128M / 16G: BAR 15 has errors during boot, BAR 13 has errors upon eGPU connection. The eGPU works.

128M / 512M: Exactly the same number and type of errors as above. The eGPU works.

128M / 256M: No I/O resource errors. But the eGPU doesn’t work; initialization fails in almost the same way as reported here: https://bbs.archlinux.org/viewtopic.php?id=261303 (Almost as if the pci= tweaks on the kernel command line had no effect.)

(By "eGPU works" I mean that it can run Folding@Home. (I haven’t tested it in other ways.) There is (I guess) a performance penalty due to the missing MMIO resources.)

Unfortunately I have no clue which of the errors are benign and which ones require further attention. But I’ll be more than to experiment of there are any suggestions.

If you sbios supports an option like ">4G MMIO" or ">4G Decoding" try enabling that.  That should make more MMIO space available.

(In reply to Alex Deucher from comment #20)
> If you sbios supports an option like ">4G MMIO" or ">4G Decoding" try
> enabling that.  That should make more MMIO space available.

I always have that enabled. (There are 64b resources in my dmesg.)