I'm utterly confused as to why the kernel decided to "xHCI host not responding to stop endpoint command".

I didn't do anything at the time. Wasn't even using the mouse.

Something funky is going on with 6.13.

This was reported in the SUSE bug tracker earlier:

https://bugzilla.suse.com/show_bug.cgi?id=1236992

I don't see it being reported here, so the issue is not new.

Yet I see no patches queued for 6.13.5.

The SUSE issue is seemingly unrelated, please dismiss.

This just happened again:

[161470.836493] PM: resume devices took 0.547 seconds
[161470.836720] OOM killer enabled.
[161470.836721] Restarting tasks ... done.
[161470.839715] random: crng reseeded on system resumption
[161470.845090] PM: suspend exit
[161471.322491] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.0: DSP1: Firmware: 400a4 vendor: 0x2 v0.43.1, 2 algorithms
[161471.324469] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.0: DSP1: cirrus/cs35l41-dsp1-spk-prot-103c8b72.bin: v0.43.1
[161471.324480] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.0: DSP1: spk-prot: D:\Amp Tuning\HP\840\0930\103C8B45_220930.bin
[161471.403951] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.1: Calibration applied: R0=10446
[161471.407392] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.0: Calibration applied: R0=10526
[161471.432157] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.1: Firmware Loaded - Type: spk-prot, Gain: 17
[161471.433916] cs35l41-hda i2c-CSC3551:00-cs35l41-hda.0: Firmware Loaded - Type: spk-prot, Gain: 17
[161471.523827] hp_wmi: Unknown event_id - 131073 - 0x0
[162644.637587] xhci_hcd 0000:c3:00.3: xHCI host not responding to stop endpoint command
[162644.651068] xhci_hcd 0000:c3:00.3: xHCI host controller not responding, assume dead
[162644.651076] xhci_hcd 0000:c3:00.3: HC died; cleaning up
[162644.651099] xhci_hcd 0000:c3:00.3: Timeout while waiting for stop endpoint command
[162644.651102] usb 1-2: USB disconnect, device number 4
[162644.678374] usb 1-3: USB disconnect, device number 2
[162644.678748] usb 1-4: USB disconnect, device number 3

Shortly after resume all the USB ports are disabled.

I'm reverting back to Linux 6.11. I cannot use my device like this.

6.13 has a lot of changes related to endpoint stopping:

e21ebe51af68 xhci: Turn NEC specific quirk for handling Stop Endpoint errors generic
474538b8dd1c usb: xhci: Avoid queuing redundant Stop Endpoint commands
484c3bab2d5d usb: xhci: Fix TD invalidation under pending Set TR Dequeue
42b758137601 usb: xhci: Limit Stop Endpoint retries

Endpoints are stopped in order to cancel transfers, before suspend, and to soft reset an endpoint after clearing a halt. 

I understand that bisecting an issue like this that triggers rarely isn't an option, but can I ask you to try running 6.13 with xhci dynamic debug enabled.

mount -t debugfs none /sys/kernel/debug
echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control
and send dmesg after issue is triggered.

It could reveal a bit more what's going on

(In reply to Mathias Nyman from comment #5)
> I understand that bisecting an issue like this that triggers rarely isn't an
> option, but can I ask you to try running 6.13 with xhci dynamic debug
> enabled.

Will do as soon as possible. Thanks a lot!

Which exact versions were you running successfully and for how long?

These patches listed by Mathias are instant first suspects, but they were all backported to v6.12.7 in December. Most of them also to v6.11.11 in early December and later in January to some LTS series.

Any chance that hibernation is indeed a (delayed) trigger and you weren't doing it as often in the past?

Did you come across similar reports from stable kernel branches in this year?

> Which exact versions were you running successfully and for how long?

Kernel 6.12.14 that I was running earlier didn't have this issue.

Used software suspend/resume multiple times successfully.

> Any chance that hibernation is indeed a (delayed) trigger and you weren't
> doing it as often in the past?

Not using hibernation, just software suspend.

I've not changed anything software-wise except installing a new kernel on this laptop.

> Did you come across similar reports from stable kernel branches in this year?

I've Googled a couple of times already for this exact error message and nothing turned up.

Created attachment 307725 [details]
xhci_hcd and usb debug log

(In reply to Mathias Nyman from comment #5)
> 6.13 has a lot of changes related to endpoint stopping:
> 
> e21ebe51af68 xhci: Turn NEC specific quirk for handling Stop Endpoint errors
> generic
> 474538b8dd1c usb: xhci: Avoid queuing redundant Stop Endpoint commands
> 484c3bab2d5d usb: xhci: Fix TD invalidation under pending Set TR Dequeue
> 42b758137601 usb: xhci: Limit Stop Endpoint retries
> 
> Endpoints are stopped in order to cancel transfers, before suspend, and to
> soft reset an endpoint after clearing a halt. 
> 
> I understand that bisecting an issue like this that triggers rarely isn't an
> option, but can I ask you to try running 6.13 with xhci dynamic debug
> enabled.
> 
> mount -t debugfs none /sys/kernel/debug
> echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
> echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control
> and send dmesg after issue is triggered.
> 
> It could reveal a bit more what's going on

I'm confused.

If I resume the laptop and don't run these three commands immediately, all the USB ports eventually die (usually under 5 minutes).

If I resume the laptop and run these commands immediately, USB ports continue working like they always did before. So, weirdly and unexpectedly, when debugging is on ... it fixes the issue.

If I resume the laptop, don't run these commands, and then when the USB ports die I run them, there are no further messages from the xhci_hcd module.

I'm attaching the debug log (again, no bug here) regardless. Maybe it contains something that will let you understand what is going on.

What if you enable only this one thing? Does anything show up under normal use or before the HC dies (if it still does)?

echo 'func xhci_handle_cmd_stop_ep +p' >/proc/dynamic_debug/control

(In reply to Artem S. Tashkinov from comment #9)
> Created attachment 307725 [details]
> xhci_hcd and usb debug log

Thanks, It does show some last minute urb canceling for endpoints
on the 1-3 device before suspend, and some more canceling after resume.
Also overflow/underflow messages after resume indicating that endpoint
might be started early.   

> 
> I'm confused.
> 
> If I resume the laptop and don't run these three commands immediately, all
> the USB ports eventually die (usually under 5 minutes).
> 
> If I resume the laptop and run these commands immediately, USB ports
> continue working like they always did before. So, weirdly and unexpectedly,
> when debugging is on ... it fixes the issue.

dynamic debug adds delays, and the code that starts and stops endpoints is
a bit timing sensitive. Could be that enabling debug hides the issue.

Can you still run one more try with xhci tracing instead of dynamic debug?
It does not affect timing as much:

mount -t debugfs none /sys/kernel/debug
echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
echo 1 > /sys/kernel/debug/tracing/tracing_on
< Reproduce issue >
Send content of /sys/kernel/debug/tracing/trace

The trace file grows fast so copy it as soon as possible after issue is triggered.

Thanks

(In reply to Michał Pecio from comment #10)
> What if you enable only this one thing? Does anything show up under normal
> use or before the HC dies (if it still does)?
> 
> echo 'func xhci_handle_cmd_stop_ep +p' >/proc/dynamic_debug/control

The same issue. Without debug the USB controller dies in a few minutes, with just this line it is working just fine.

> Can you still run one more try with xhci tracing instead of dynamic debug?
It does not affect timing as much:

I will try this ASAP.

Hmm, can enabling a debug message which never gets printed really make a big difference? Maybe I'm crazy, but I wonder if it's possible that your interaction with USB devices while entering those commands somehow prevents (or at least delays) the failure?

I suppose that both dynamic debug and tracing settings persist across a suspend cycle, so if you still have no luck, maybe try setting everything up before suspending and then do nothing after resuming?

I think I found it.

Does it help if you revert 36b972d4b7cef?

I found this issue happening this after resuming from sleep. Downgrading to Kernel 6.12.9 (6.12.9-arch1-1) removed the issue completely. The error message was the same.

I am not much knowledgeable in reporting or helping with kernel issues but if you give me instructions of what I can do to maybe test a patch and report back I can help.

(In reply to bugzilla from comment #15)
> I found this issue happening this after resuming from sleep. Downgrading to
> Kernel 6.12.9 (6.12.9-arch1-1) removed the issue completely. The error
> message was the same.
> 
> I am not much knowledgeable in reporting or helping with kernel issues but
> if you give me instructions of what I can do to maybe test a patch and
> report back I can help.

Please follow the instructions in https://bugzilla.kernel.org/show_bug.cgi?id=219824#c11

I'm currently on vacation, I left my mouse at home, so I cannot debug this.

BTW, here's an interesting tidbit: even with debugging enabled in comment 11 I wasn't able to get my USB ports die but (!) on a second resume they die _immediately_. So, maybe I just had to suspend/resume twice and then generate the debug data. Sadly, I just ran out of time.

Also, if you're able to compile your kernel, please try to revert commit 36b972d4b7cef on top of e.g. 6.13.5 and check it you can reproduce the issue.

The patch can be downloaded here: https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=36b972d4b7cef5d098de63fee8d00720c051f335

You'll need to `patch -R < patch` it.

Looks like there are multiple people affected by this regression, reports have started to trickle in:

https://bbs.archlinux.org/viewtopic.php?pid=2229550

BTW https://lore.kernel.org/lkml/20250304085139.4610e8ff@foxbook/ has been queued for stable, looks like 6.13.6 will have the issue fixed.

Please check.

(In reply to Michał Pecio from comment #14)
> I think I found it.
> 
> Does it help if you revert 36b972d4b7cef?

People claim it fixes the issue.

Please actually submit the revert to stable as 6.13.6 has been released without it and neither 6.14-rc has seemingly seen it.

And seeing "Reported by:" would be nice. Thanks.

Sorry about Reported-by, but if I added that then they would also want a Closes and I wasn't sure if it really is this bug. It's something I produced on my machine under my workload, using knowledge of how one commit is broken.

I guess I could have taken a gamble and tagged it anyway because it seemed likely, but I had no confirmation and no idea if I would get any by the end of the week. Indeed, to this day the best I've seen is reports that downgrading to 6.12 helps, I haven't heard from anyone previously affected reverting the suspect patch.

Like Mathias, I was expecting some nightmare scenario and not that. I hope that this is all there was to it, but we will really know next week.

My patch made it to Greg's usb-linus branch, which normally means it will land in the next -rc tomorrow and then propagate to stable. No other -rc has it because they were released before it existed.

Will this be backported to 6.13? Seem 6.13.7 doesn't have it.

6.13.7 absolutely includes it:

https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.13.7

> commit 80cb8e694110dee4ac6fbf0956ba7439aeb0603d
> Author: Michal Pecio <michal.pecio@gmail.com>
> Date:   Tue Mar 4 13:31:47 2025 +0200
> 
>     usb: xhci: Fix host controllers "dying" after suspend and resume
>     
>     commit c7c1f3b05c67173f462d73d301d572b3f9e57e3b upstream.
>     
>     A recent cleanup went a bit too far and dropped clearing the cycle bit
>     of link TRBs, so it stays different from the rest of the ring half of
>     the time. Then a race occurs: if the xHC reaches such link TRB before
>     more commands are queued, the link's cycle bit unintentionally matches
>     the xHC's cycle so it follows the link and waits for further commands.
>     If more commands are queued before the xHC gets there, inc_enq() flips
>     the bit so the xHC later sees a mismatch and stops executing commands.
>     
>     This function is called before suspend and 50% of times after resuming
>     the xHC is doomed to get stuck sooner or later. Then some Stop Endpoint
>     command fails to complete in 5 seconds and this shows up
>     
>     xhci_hcd 0000:00:10.0: xHCI host not responding to stop endpoint command
>     xhci_hcd 0000:00:10.0: xHCI host controller not responding, assume dead
>     xhci_hcd 0000:00:10.0: HC died; cleaning up
>     
>     followed by loss of all USB decives on the affected bus. That's if you
>     are lucky, because if Set Deq gets stuck instead, the failure is silent.
>     
>     Likely responsible for kernel bug 219824. I found this while searching
>     for possible causes of that regression and reproduced it locally before
>     hearing back from the reporter. To repro, simply wait for link cycle to
>     become set (debugfs), then suspend, resume and wait. To accelerate the
>     failure I used a script which repeatedly starts and stops a UVC camera.
>     
>     Some HCs get fully reinitialized on resume and they are not affected.

(In reply to Artem S. Tashkinov from comment #22)
> 6.13.7 absolutely includes it:

> https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.13.7


Ah sorry, my bad. Thanks!