Bug 217745

Summary:	Ubuntu 22.04, "nvme list" will hang for 60 seconds after "nvme subsystem-reset"
Product:	Drivers	Reporter:	shenlinghai (shenlinghai)
Component:	Flash/Memory Technology Devices	Assignee:	David Woodhouse (dwmw2)
Status:	NEW ---
Severity:	normal	CC:	bagasdotme, regressions
Priority:	P3
Hardware:	Intel
OS:	Linux
Kernel Version:		Subsystem:
Regression:	No	Bisected commit-id:

Description shenlinghai 2023-08-01 11:38:16 UTC

Issue description:
root:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy

root:~# uname -a
Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

root:~#nvme subsystem-reset /dev/nvme0

root:~#nvme list      //This command will hang about 60 seconds. 

Issue analysis:
Maybe it hangs in function nvme_wait_reset. When we revert the following commit, this issue will disappear.
if (!nvme_wait_reset(ctrl))
https://github.com/torvalds/linux/commit/1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split

Comment 1 Bagas Sanjaya 2023-08-02 11:28:41 UTC

(In reply to shenlinghai from comment #0)
> Issue description:
> root:~# lsb_release -a
> No LSB modules are available.
> Distributor ID:       Ubuntu
> Description:  Ubuntu 22.04 LTS
> Release:      22.04
> Codename:     jammy
> 
> root:~# uname -a
> Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> 
> root:~#nvme subsystem-reset /dev/nvme0
> 
> root:~#nvme list      //This command will hang about 60 seconds. 
> 
> Issue analysis:
> Maybe it hangs in function nvme_wait_reset. When we revert the following
> commit, this issue will disappear.
> if (!nvme_wait_reset(ctrl))
> https://github.com/torvalds/linux/commit/
> 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split

Can you check latest mainline (currently v6.5-rc4)?

Comment 2 Bagas Sanjaya 2023-08-02 11:38:23 UTC

(In reply to shenlinghai from comment #0)
> Issue description:
> root:~# lsb_release -a
> No LSB modules are available.
> Distributor ID:       Ubuntu
> Description:  Ubuntu 22.04 LTS
> Release:      22.04
> Codename:     jammy
> 
> root:~# uname -a
> Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> 
> root:~#nvme subsystem-reset /dev/nvme0
> 
> root:~#nvme list      //This command will hang about 60 seconds. 
> 
> Issue analysis:
> Maybe it hangs in function nvme_wait_reset. When we revert the following
> commit, this issue will disappear.
> if (!nvme_wait_reset(ctrl))
> https://github.com/torvalds/linux/commit/
> 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split

Also, can you show dmesg and lspci output?

Comment 3 shenlinghai 2023-08-03 04:55:35 UTC

(In reply to Bagas Sanjaya from comment #2)
> (In reply to shenlinghai from comment #0)
> > Issue description:
> > root:~# lsb_release -a
> > No LSB modules are available.
> > Distributor ID:       Ubuntu
> > Description:  Ubuntu 22.04 LTS
> > Release:      22.04
> > Codename:     jammy
> > 
> > root:~# uname -a
> > Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> > PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > root:~#nvme subsystem-reset /dev/nvme0
> > 
> > root:~#nvme list      //This command will hang about 60 seconds. 
> > 
> > Issue analysis:
> > Maybe it hangs in function nvme_wait_reset. When we revert the following
> > commit, this issue will disappear.
> > if (!nvme_wait_reset(ctrl))
> > https://github.com/torvalds/linux/commit/
> > 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split
> 
> Also, can you show dmesg and lspci output?

Here is the dmesg. 
nvme subsytem-reset /dev/nvme0 at 11:51:03. And then execute nvme list.
And the nvme0 is normal at 11:52:05.


[11:51:03 2023] pcieport 0000:80:01.1: pciehp: Slot(17): Link Down
[11:52:04 2023] nvme 0000:81:00.0: Unable to change power state from D3cold to D0, device inaccessible
[11:52:04 2023] nvme nvme0: Removing after probe failure status: -19
[11:52:04 2023] nvme0n1: detected capacity change from 27346862080 to 0
[11:52:04 2023] pci 0000:81:00.0: Removing from iommu group 10
[11:52:05 2023] pcieport 0000:80:01.1: pciehp: Slot(17): Card present
[11:52:05 2023] pcieport 0000:80:01.1: pciehp: Slot(17): Link Up
[11:52:05 2023] pci 0000:81:00.0: [1e3b:0600] type 00 class 0x010802
[11:52:05 2023] pci 0000:81:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[11:52:05 2023] pci 0000:81:00.0: reg 0x20: [mem 0x00000000-0x00001fff 64bit pref]
[11:52:05 2023] pci 0000:81:00.0: reg 0x30: [mem 0x00000000-0x00007fff pref]
[11:52:05 2023] pci 0000:81:00.0: Max Payload Size set to 512 (was 128, max 512)
[11:52:05 2023] pci 0000:81:00.0: Adding to iommu group 10
[11:52:05 2023] pcieport 0000:80:01.1: bridge window [io  0x1000-0x0fff] to [bus 81] add_size 1000
[11:52:05 2023] pcieport 0000:80:01.1: BAR 13: no space for [io  size 0x1000]
[11:52:05 2023] pcieport 0000:80:01.1: BAR 13: failed to assign [io  size 0x1000]
[11:52:05 2023] pcieport 0000:80:01.1: BAR 13: no space for [io  size 0x1000]
[11:52:05 2023] pcieport 0000:80:01.1: BAR 13: failed to assign [io  size 0x1000]
[11:52:05 2023] pci 0000:81:00.0: BAR 0: assigned [mem 0xb2400000-0xb240ffff 64bit]
[11:52:05 2023] pci 0000:81:00.0: BAR 6: assigned [mem 0xb2410000-0xb2417fff pref]
[11:52:05 2023] pci 0000:81:00.0: BAR 4: assigned [mem 0x58140f00000-0x58140f01fff 64bit pref]
[11:52:05 2023] pcieport 0000:80:01.1: PCI bridge to [bus 81]
[11:52:05 2023] pcieport 0000:80:01.1:   bridge window [mem 0xb2400000-0xb24fffff]
[11:52:05 2023] pcieport 0000:80:01.1:   bridge window [mem 0x58140f00000-0x58140ffffff 64bit pref]
[11:52:05 2023] nvme nvme0: pci function 0000:81:00.0
[11:52:05 2023] nvme 0000:81:00.0: enabling device (0000 -> 0002)
[11:52:05 2023] nvme nvme0: 64/0/0 default/read/poll queues
[11:52:05 2023] nvme_log_error: 20 callbacks suppressed

Comment 4 shenlinghai 2023-08-03 04:57:37 UTC

(In reply to Bagas Sanjaya from comment #1)
> (In reply to shenlinghai from comment #0)
> > Issue description:
> > root:~# lsb_release -a
> > No LSB modules are available.
> > Distributor ID:       Ubuntu
> > Description:  Ubuntu 22.04 LTS
> > Release:      22.04
> > Codename:     jammy
> > 
> > root:~# uname -a
> > Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> > PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > root:~#nvme subsystem-reset /dev/nvme0
> > 
> > root:~#nvme list      //This command will hang about 60 seconds. 
> > 
> > Issue analysis:
> > Maybe it hangs in function nvme_wait_reset. When we revert the following
> > commit, this issue will disappear.
> > if (!nvme_wait_reset(ctrl))
> > https://github.com/torvalds/linux/commit/
> > 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split
> 
> Can you check latest mainline (currently v6.5-rc4)?

OK. I'll try. I'm not sure if nvme driver v6.5 can be installed in Ubuntu 22.04.

Comment 5 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-08-07 10:21:01 UTC

(In reply to shenlinghai from comment #4)

> > Can you check latest mainline (currently v6.5-rc4)?
> OK. I'll try. I'm not sure if nvme driver v6.5 can be installed in Ubuntu
> 22.04.

FWIW, you should not try just the 6.5 nvme code, you should try a proper 6.5-rc5 kenrel -- and that almost always should work (at least from the kernel side and if the configs  are similar).

Comment 6 Bagas Sanjaya 2023-08-29 08:24:01 UTC

(In reply to shenlinghai from comment #4)
> (In reply to Bagas Sanjaya from comment #1)
> > (In reply to shenlinghai from comment #0)
> > > Issue description:
> > > root:~# lsb_release -a
> > > No LSB modules are available.
> > > Distributor ID:       Ubuntu
> > > Description:  Ubuntu 22.04 LTS
> > > Release:      22.04
> > > Codename:     jammy
> > > 
> > > root:~# uname -a
> > > Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> > > PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > > 
> > > root:~#nvme subsystem-reset /dev/nvme0
> > > 
> > > root:~#nvme list      //This command will hang about 60 seconds. 
> > > 
> > > Issue analysis:
> > > Maybe it hangs in function nvme_wait_reset. When we revert the following
> > > commit, this issue will disappear.
> > > if (!nvme_wait_reset(ctrl))
> > > https://github.com/torvalds/linux/commit/
> > > 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split
> > 
> > Can you check latest mainline (currently v6.5-rc4)?
> 
> OK. I'll try. I'm not sure if nvme driver v6.5 can be installed in Ubuntu
> 22.04.

shenlinghai, any news? Where is your test results?

Comment 7 shenlinghai 2023-08-30 06:35:54 UTC

(In reply to Bagas Sanjaya from comment #6)
> (In reply to shenlinghai from comment #4)
> > (In reply to Bagas Sanjaya from comment #1)
> > > (In reply to shenlinghai from comment #0)
> > > > Issue description:
> > > > root:~# lsb_release -a
> > > > No LSB modules are available.
> > > > Distributor ID:       Ubuntu
> > > > Description:  Ubuntu 22.04 LTS
> > > > Release:      22.04
> > > > Codename:     jammy
> > > > 
> > > > root:~# uname -a
> > > > Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> > > > PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64
> GNU/Linux
> > > > 
> > > > root:~#nvme subsystem-reset /dev/nvme0
> > > > 
> > > > root:~#nvme list      //This command will hang about 60 seconds. 
> > > > 
> > > > Issue analysis:
> > > > Maybe it hangs in function nvme_wait_reset. When we revert the
> following
> > > > commit, this issue will disappear.
> > > > if (!nvme_wait_reset(ctrl))
> > > > https://github.com/torvalds/linux/commit/
> > > > 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split
> > > 
> > > Can you check latest mainline (currently v6.5-rc4)?
> > 
> > OK. I'll try. I'm not sure if nvme driver v6.5 can be installed in Ubuntu
> > 22.04.
> 
> shenlinghai, any news? Where is your test results?

I'm not able to use nvme driver v6.5 at Ubuntu 22.04. I noted that Ubuntu 23.10 that will be released in Oct. will integrate linux kernel v6.5.

Comment 8 shenlinghai 2023-08-31 03:33:52 UTC

(In reply to Bagas Sanjaya from comment #6)
> (In reply to shenlinghai from comment #4)
> > (In reply to Bagas Sanjaya from comment #1)
> > > (In reply to shenlinghai from comment #0)
> > > > Issue description:
> > > > root:~# lsb_release -a
> > > > No LSB modules are available.
> > > > Distributor ID:       Ubuntu
> > > > Description:  Ubuntu 22.04 LTS
> > > > Release:      22.04
> > > > Codename:     jammy
> > > > 
> > > > root:~# uname -a
> > > > Linux dapu-Star-Lake 5.19.0-46-generic #47~22.04.1-Ubuntu SMP
> > > > PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64
> GNU/Linux
> > > > 
> > > > root:~#nvme subsystem-reset /dev/nvme0
> > > > 
> > > > root:~#nvme list      //This command will hang about 60 seconds. 
> > > > 
> > > > Issue analysis:
> > > > Maybe it hangs in function nvme_wait_reset. When we revert the
> following
> > > > commit, this issue will disappear.
> > > > if (!nvme_wait_reset(ctrl))
> > > > https://github.com/torvalds/linux/commit/
> > > > 1e866afd4bcdd01a70a5eddb4371158d3035ce03?diff=split
> > > 
> > > Can you check latest mainline (currently v6.5-rc4)?
> > 
> > OK. I'll try. I'm not sure if nvme driver v6.5 can be installed in Ubuntu
> > 22.04.
> 
> shenlinghai, any news? Where is your test results?

This bug exits in Ubuntu 6.2.0-26. This week my server's linux kernel was automatically updated to v6.2.
Linux dapu-Star-Lake 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Comment 9 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-09-05 14:14:56 UTC

this bug tracker is for the upstream kernel; if you want help with ubuntu kernels, you have to report this to the ubuntu project; for a more details explanation see
https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kernel-bug-reports-are-ignored/#your-kernel-was-apparently-built-by-a-linux-distributor-or-hardware-vendor

Comment 10 shenlinghai 2023-09-06 02:58:35 UTC

(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #9)
> this bug tracker is for the upstream kernel; if you want help with ubuntu
> kernels, you have to report this to the ubuntu project; for a more details
> explanation see
> https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-
> kernel-bug-reports-are-ignored/#your-kernel-was-apparently-built-by-a-linux-
> distributor-or-hardware-vendor

OK. I have repoted this to the Ubuntu project.