Created attachment 300914 [details]
lspci after boot (no EGPU connected)

Created attachment 300915 [details]
lspci after attaching EGPU

Created attachment 300916 [details]
lspci after detaching EGPU

From Message-ID <23d4b1f4-09a7-c12a-7610-2863c8267341@yahoo.it>, unfortunately not archived because it contains HTML.

On 09/05/22 18:23, Bjorn Helgaas wrote:
> On Sun, May 8, 2022 at 3:29 PM <bugzilla-daemon@kernel.org> wrote:
>> https://bugzilla.kernel.org/show_bug.cgi?id=215958
>>
>>             Bug ID: 215958
>>            Summary: thunderbolt3 egpu cannot disconnect cleanly
>>            Product: Drivers
>>            Version: 2.5
>>     Kernel Version: 5.17.0-1003-oem #3-Ubuntu SMP PREEMPT
>>           Hardware: All
>>                 OS: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: PCI
>>           Assignee: drivers_pci@kernel-bugs.osdl.org
>>           Reporter: r087r70@yahoo.it
>>         Regression: No
> I assume this is not a regression, right?  If it is a regression, what
> previous kernel worked correctly?

no it's not, but I haven't tested with all the possible kernel versions, just with 5.15 and 5.17

>> I have an external egpu (Radeon 6600 RX) connected through thunderbolt3 to
>> my
>> Thinkpad X1 carbon 6th Gen.. When I disconnect the thunderbolt3 cable I get
>> the
>> following error in dmesg:
>>
>> [21874.194994] amdgpu 0000:0c:00.0: amdgpu: SMU: response:0xFFFFFFFF for
>> index:18 param:0x00000005 message:TransferTableSmu2Dram?
>> ...
>> ...
>> [21879.544226] amdgpu 0000:0c:00.0: amdgpu: Failed to disable smu features.
>> [21879.544230] amdgpu 0000:0c:00.0: amdgpu: Fail to disable dpm features!
>> [21879.544238] [drm] free PSP TMR buffer
> The above looks like what amdgpu would see when the GPU is no longer
> accessible (writes are dropped and reads return 0xffffffff). It's
> possible amdgpu could notice this and shut down more gracefully, but I
> don't think it's the main problem here and it probably wouldn't force
> you to reboot.

actually in this state I cannot `modprobe -r amdgpu`:

modprobe: FATAL: Module amdgpu is in use.



>> [21880.455935] i915 0000:00:02.0: vgaarb: changed VGA decodes:
>> olddecodes=none,decodes=io+mem:owns=io+mem
>> [21880.456218] pci 0000:0c:00.0: Removing from iommu group 14
>> ...
>> ...
>> [21880.457311] pci_bus 0000:09: busn_res: [bus 09-3a] is released
>> [21880.457543] pci 0000:08:00.0: Removing from iommu group 14
> This looks like removing 0c:00.0 (the GPU) and two switches leading to
> it (probably part of the Thunderbolt topology), so to be expected.
>
>> [21880.457847] pci_bus 0000:06: Allocating resources
>> [21880.457888] pcieport 0000:06:02.0: bridge window [io 0x1000-0x0fff] to
>> [bus
>> 3b] add_size 1000
>> ...
>> ...
>> [21880.457947] pcieport 0000:06:02.0: BAR 13: failed to assign [io  size
>> 0x1000]
> I'm not sure why we're allocating resources as part of the removal.
> The hierarchies under 06:02.0 (to [bus 3b]) and 06:04.0 (to [bus
> 3c-6f]) seem to be siblings of the hierarchy you just removed (my
> guess is that was 06:01.0 to [bus 08-3a]).  But again, shouldn't
> require a reboot.
>
>> upon reconnection of the cable I get:
>>
>> [22192.753261] input: HDA ATI HDMI HDMI/DP,pcm=3 as
>>
>> /devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:01.0/0000:08:00.0/0000:09:01.0/0000:0a:00.0/0000:0b:00.0/0000:0c:00.1/sound/card1/input98
>> [22192.753738] input: HDA ATI HDMI HDMI/DP,pcm=7 as
>>
>> /devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:01.0/0000:08:00.0/0000:09:01.0/0000:0a:00.0/0000:0b:00.0/0000:0c:00.1/sound/card1/input99
>> [22192.753952] input: HDA ATI HDMI HDMI/DP,pcm=8 as
>>
>> /devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:01.0/0000:08:00.0/0000:09:01.0/0000:0a:00.0/0000:0b:00.0/0000:0c:00.1/sound/card1/input100
>> [22192.755234] input: HDA ATI HDMI HDMI/DP,pcm=9 as
>>
>> /devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:01.0/0000:08:00.0/0000:09:01.0/0000:0a:00.0/0000:0b:00.0/0000:0c:00.1/sound/card1/input101
>> [22192.763885] input: HDA ATI HDMI HDMI/DP,pcm=10 as
>>
>> /devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:01.0/0000:08:00.0/0000:09:01.0/0000:0a:00.0/0000:0b:00.0/0000:0c:00.1/sound/card1/input102
>> [22192.975773] thunderbolt 0-1: new device found, vendor=0x127 device=0x1
>> [22192.975786] thunderbolt 0-1: Razer Core X
>>
>> but the egpu no longer appears in `xrandr --listproviders`. Full reboot is
>> needed.
> Can you please build with CONFIG_DYNAMIC_DEBUG=y, boot with
> 'dyndbg="file pciehp* +p"', and attach the complete dmesg log to the
> bugzilla?  Also please attach the complete "sudo lspci -vv" output
> (before the unplug and after the replug)?

Ironically, I have rebooted to get the lspci output,  and now I can no longer get into the above state. What I get is that after attaching the egpu, it is enabled *without* the need of restarting the Xserver, while after detaching it the Xserver is restarted and the card gets released correctly, although the amdgpu drivers stays loaded. But I can `modprobe -r amdgpu` without problems. I could connect/disconnect many time without issues. Attached is the lspci output after fresh boot, upon epgu connection, and disconnection.

I will test more in the next days.

Thank you,
Roberto

So is there any problem currently ? Also please provide full dmesg log for both disconnect and recconect of eGPU.

So is there any problem currently ? Also please provide full dmesg log for both disconnect and recconect of eGPU.

Please give me some time, I'll be quite overloaded until this weekend, I will write back asap.

Created attachment 300955 [details]
dmesg during suspend and resume (ending in system freeze)

I notice two major issues related to the present bug:

1) suspend-resume seems to trigger the bug in subject (one can no longer connect/disconnect the card cleanly)

2) upon disconnection of the egpu, the X server restarts even if it's not using it: Note that I only use the egpu indirectly through DRI_PRIME offloading, and thus there should be no need of restarting X server, which is instead using the integrated "Intel UHD Graphics 620". The restart of X server is very annoying.

I have attached the `sudo dmesg -ew > dmesg.log` during suspend and resume operations, which ended in an unresponsive system (freezed with white screen).
I have set 'dyndbg="file pciehp* +p"' in grub at boot.

Cn you confirm you have no problems with disconnecting eGPU as long as you don't do susepnd/resume ? 

It seems your susepnd/resume is broken regardless of eGPU disconnection procedure so it all looks more like S3 bug which has nothing to do with eGPU disconnect. 

Can you take you graphic card out of eGPU, put it on the motherboard PCI bus and check how suspend/resume works for you ?

(In reply to Andrey Grodzovsky from comment #10)
> Cn you confirm you have no problems with disconnecting eGPU as long as you
> don't do susepnd/resume?

as already stated, the only problem is that X server restarts, which I strongly preferred it didn't

> It seems your susepnd/resume is broken regardless of eGPU disconnection
> procedure so it all looks more like S3 bug which has nothing to do with eGPU
> disconnect. 

I see. Should we redirect this bug to the ACPI area?

> Can you take you graphic card out of eGPU, put it on the motherboard PCI bus
> and check how suspend/resume works for you ?

Unfortunately I cannot, I only have a laptop not a desktop.

(In reply to Roberto from comment #11)
> (In reply to Andrey Grodzovsky from comment #10)
> > Cn you confirm you have no problems with disconnecting eGPU as long as you
> > don't do susepnd/resume?
> 
> as already stated, the only problem is that X server restarts, which I
> strongly preferred it didn't


Please open a seperate ticket for this and attach both dmesg and Xorg.log for this. 

> 
> > It seems your susepnd/resume is broken regardless of eGPU disconnection
> > procedure so it all looks more like S3 bug which has nothing to do with
> eGPU
> > disconnect. 
> 
> I see. Should we redirect this bug to the ACPI area?

No, it's our responsibility too. Can you go back and forth a bit and see if this is a regression ? Try maybe 5.15 and 5.16 kernels and see if this problem still there.


> 
> > Can you take you graphic card out of eGPU, put it on the motherboard PCI
> bus
> > and check how suspend/resume works for you ?
> 
> Unfortunately I cannot, I only have a laptop not a desktop.

Created attachment 300999 [details]
dmesg during suspend and resume using kernel 5.15.0

Using kernel 5.15 I could suspend, but during resume it seemed to froze for a few seconds, but then it managed to resume. I have tried to reconnect the egpu and it seemed to work, although with some lagging: if I type glxinfo or glxgears it takes a couple of seconds to respond.

Moreover, as second issue, for both the kernels I cannot use the `sensors` application to monitor the temperature after resuming from suspend, since it hangs. If I type 'sensors' in a terminal it prints ~40 lines with some information about the cpu and the battery, but then it freezes and it does not exit.

The suspend/resume procedure seems really problematic with the egpu attached, or after detaching it.

(In reply to Roberto from comment #14)
> Using kernel 5.15 I could suspend, but during resume it seemed to froze for
> a few seconds, but then it managed to resume. I have tried to reconnect the
> egpu and it seemed to work, although with some lagging: if I type glxinfo or
> glxgears it takes a couple of seconds to respond.
> 
> Moreover, as second issue, for both the kernels I cannot use the `sensors`
> application to monitor the temperature after resuming from suspend, since it
> hangs. If I type 'sensors' in a terminal it prints ~40 lines with some
> information about the cpu and the battery, but then it freezes and it does
> not exit.
> 
> The suspend/resume procedure seems really problematic with the egpu
> attached, or after detaching it.

Probably, we don't have porpoer eGPU cage at hand to test it each time. Can you try to bisect from this 5.15 kernel forward to see at what point it started hard hanging like in your original kernel ?

For the Xserves issue I've created https://bugzilla.kernel.org/show_bug.cgi?id=216004