Bug 214771

Summary: controller reset and subsystem reset cause Kernel panic
Product: IO/Storage Reporter: LiuZhouhua (liuzhouhua)
Component: NVMeAssignee: IO/NVME Virtual Default Assignee (io_nvme)
Status: NEW ---    
Severity: enhancement CC: carnil, jch, kbusch, sbeattie
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 4.19 Subsystem:
Regression: No Bisected commit-id:
Attachments: proposed fix

Description LiuZhouhua 2021-10-20 13:23:30 UTC 
Massive sending NVME_IOCTL_RESET and NVME_IOCTL_SUBSYS_RESET by user tool at the same time will cause OS panic. Mybe the following modifications in nvme_dev_ioctl can make driver stronger.

@@ -3031,15 +3031,23 @@ static long nvme_dev_ioctl(struct file *
 	struct nvme_ctrl *ctrl = file->private_data;
 	void __user *argp = (void __user *)arg;
 
+    if (ctrl == NULL)
+        return -ENOTTY;
+    if (ctrl->state == NVME_CTRL_DELETING || ctrl->state == NVME_CTRL_DEAD)
+        return -ENOTTY;
+


[361245.754472] nvme nvme2: resetting controller
[361245.754500] nvme nvme2: resetting controller
[361245.761914] nvme nvme2: resetting controller
[361245.762081] nvme nvme2: resetting controller
[361245.762136] nvme nvme2: resetting controller
[361245.762246] Unable to handle kernel paging request at virtual address ffff000081c42020
[361245.762247] Mem abort info:
[361245.762248]   ESR = 0x96000047
[361245.762250]   Exception class = DABT (current EL), IL = 32 bits
[361245.762251]   SET = 0, FnV = 0
[361245.762251]   EA = 0, S1PTW = 0
[361245.762252] Data abort info:
[361245.762253]   ISV = 0, ISS = 0x00000047
[361245.762254]   CM = 0, WnR = 1
[361245.762256] swapper pgtable: 4k pages, 48-bit VAs, pgdp = 00000000c96d3c05
[361245.762258] [ffff000081c42020] pgd=00002027ffffe003, pud=00002027ffffd003, pmd=00000027df6be003, pte=0000000000000000
[361245.762263] Internal error: Oops: 96000047 [#1] SMP
[361245.767407] Process trinity-c383 (pid: 2157089, stack limit = 0xffff000099ea8000)
[361245.775244] CPU: 95 PID: 2157089 Comm: trinity-c383 Kdump: loaded Tainted: G           OE     4.19.90-vhulk2011.1.0.h352.eulerosv2r9.aarch64 #1
[361245.788669] Hardware name: Huawei TaiShan 2280 V2/BC82AMDDA, BIOS 1.08 09/15/2019
[361245.796505] pstate: 60400009 (nZCv daif +PAN -UAO)
[361245.801548] pc : nvme_pci_reg_write32+0x30/0x48 [nvme]
[361245.806954] lr : nvme_dev_ioctl+0x74/0x1f0 [nvme_core]
[361245.812353] sp : ffff000099eabd00
[361245.815856] x29: ffff000099eabd00 x28: ffff80277c503e00 
[361245.821434] x27: 0000000000000000 x26: 0000000000000000 
[361245.827009] x25: 0000000056000000 x24: ffff8027811fdd68 
[361245.832587] x23: ffff8027d26e2700 x22: 0000000000000185 
[361245.838164] x21: ffffa0277d04e238 x20: 000000004e564d65 
[361245.843740] x19: ffff000081c42020 x18: 0000000000000000 
[361245.849315] x17: 0000000000000000 x16: 0000000000000000 
[361245.854891] x15: 0000000000000000 x14: 0000000000000000 
[361245.860467] x13: 0000000000000000 x12: 0000000000000000 
[361245.866043] x11: 0000000000000000 x10: 0000000000000000 
[361245.871620] x9 : 0000000000000000 x8 : 0000000000000000 
[361245.877197] x7 : ffff000099eabda8 x6 : 0000000000000000 
[361245.882774] x5 : 0000000000000000 x4 : 0000000000000000 
[361245.888350] x3 : ffff000000b180d0 x2 : 000000004e564d65 
[361245.893929] x1 : ffff000081c42000 x0 : ffff000000aaedcc 
[361245.899508] Call trace:
[361245.906638]  nvme_pci_reg_write32+0x30/0x48 [nvme]
[361245.916366]  nvme_dev_ioctl+0x74/0x1f0 [nvme_core]
[361245.925984]  do_vfs_ioctl+0xc4/0x8c0
[361245.934237]  ksys_ioctl+0x8c/0xa0
[361245.942028]  __arm64_sys_ioctl+0x28/0x98
[361245.950291]  el0_svc_common+0x80/0x1b8
[361245.958202]  el0_svc_handler+0x78/0xe0
[361245.965904]  el0_svc+0x10/0x260
[361245.972778] Code: d503201f d5033e9f f85682a1 8b334033 (b9000274) 
[361245.982651] kernel fault(0x1) notification starting on CPU 95
[361245.992186] kernel fault(0x1) notification finished on CPU 95
[361246.001591] [kbox] catch die event on cpu 95.
[361246.009623] [kbox] catch die event, start logging.
[361246.018015] [kbox] die info:Oops:0047.
[361246.025366] [kbox] start to collect.
[361246.033241] [kbox] collected_len = 176005, g_log_buf_len_local = 1048576
[361246.043774] [kbox] save kbox pre log success.
[361246.052027] [kbox] stack:
[361246.058383] [kbox] stack grow form: 0xffff000099eabd00 to 0xffff000099eac000.
[361246.073378] [kbox] bd00: ffff000099eabd30 ffff000000aaedcc 0000000000004e45 ffffa0277d04e238
[361246.090743] [kbox] bd20: 0000000000000074 0000000000004e45 ffff000099eabd70 ffff00008038ec2c
[361246.109144] [kbox] bd40: ffff00008128a000 0000000000004e45 0000000000000074 ffff00008045ee8c
[361246.128431] [kbox] bd60: ffff000081316880 0000000000000074 ffff000099eabe00 ffff00008038f4b4
[361246.148516] [kbox] bd80: ffff8027d26e2700 0000000000000000 ffff8027d26e2700 0000000000000185
[361246.169300] [kbox] bda0: 0000000000004e45 0000000000000074 ffff000099eabd90 ffff0000801f5bc0
[361246.190529] [kbox] bdc0: ffff000099eabde0 2004bb3357eeb400 ffff000099eabe00 ffff00008038f474
[361246.212439] [kbox] bde0: ffff8027d26e2700 ffff000099eabec0 ffff8027d26e2700 2004bb3357eeb400
[361246.235176] [kbox] be00: ffff000099eabe40 ffff00008038f4f0 ffff000099eabec0 ffff000099eabec0
[361246.258513] [kbox] be20: ffffffffffffffff 0000000000000200 ffff000080aa1698 0000000000000015
[361246.282326] [kbox] be40: ffff000099eabe60 ffff000080099e98 000000000000001d ffff000080099e98
[361246.306913] [kbox] be60: ffff000099eabea0 ffff00008009a048 ffff000099eabec0 0000a0275e7d3000
[361246.332091] [kbox] be80: 00000000ffffffff 0000ffff97339614 0000000040000000 ffff000080083c0c
[361246.357695] [kbox] bea0: ffff000099eabff0 ffff000080084250 0000000000000200 ffff000080084148
Comment 1 Keith Busch 2021-10-20 14:05:59 UTC 
I think what you're showing is the driver accessing a pci register before it has iomapped it. i'll see if there's something we can do about it.
Comment 2 Keith Busch 2021-10-20 14:59:34 UTC 
Created attachment 299261 [details]
proposed fix

Could you confirm if the attached patch fixes your observation?
Comment 3 LiuZhouhua 2021-10-21 07:23:45 UTC 
(In reply to Keith Busch from comment #2)
> Created attachment 299261 [details]
> proposed fix
> 
> Could you confirm if the attached patch fixes your observation?

I've tried this patch. It hung when insmod nvme.ko

[Thu Oct 21 15:59:06 2021] INFO: task kworker/u193:2:67453 blocked for more than 120 seconds.
[Thu Oct 21 15:59:06 2021]       Tainted: G           OE     4.19.36-vhulk1907.1.0.h748.eulerosv2r8.aarch64 #1
[Thu Oct 21 15:59:06 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Oct 21 15:59:06 2021] kworker/u193:2  D    0 67453      2 0x00000228
[Thu Oct 21 15:59:06 2021] Workqueue: nvme-reset-wq nvme_reset_work [nvme]
[Thu Oct 21 15:59:06 2021] Call trace:
[Thu Oct 21 15:59:06 2021]  __switch_to+0x94/0xe8
[Thu Oct 21 15:59:06 2021]  __schedule+0x31c/0x9c8
[Thu Oct 21 15:59:06 2021]  schedule+0x2c/0x88
[Thu Oct 21 15:59:06 2021]  schedule_preempt_disabled+0x14/0x20
[Thu Oct 21 15:59:06 2021]  __mutex_lock.isra.1+0x1fc/0x540
[Thu Oct 21 15:59:06 2021]  __mutex_lock_slowpath+0x24/0x30
[Thu Oct 21 15:59:06 2021]  mutex_lock+0x80/0xa8
[Thu Oct 21 15:59:06 2021]  nvme_pci_reg_write32+0x3c/0x78 [nvme]
[Thu Oct 21 15:59:06 2021]  nvme_disable_ctrl+0x40/0x78 [nvme]
[Thu Oct 21 15:59:06 2021]  nvme_reset_work+0x260/0xea8 [nvme]
[Thu Oct 21 15:59:06 2021]  process_one_work+0x1b4/0x3f8
[Thu Oct 21 15:59:06 2021]  worker_thread+0x210/0x470
[Thu Oct 21 15:59:06 2021]  kthread+0x134/0x138
[Thu Oct 21 15:59:06 2021]  ret_from_fork+0x10/0x18
Comment 4 Keith Busch 2021-10-21 15:53:59 UTC 
Oops, of course it's a circular lock... will rework.
Comment 5 John Haxby 2022-11-09 12:54:24 UTC 
(In reply to Keith Busch from comment #4)
> Oops, of course it's a circular lock... will rework.

What happened to that?  The most recent set of changes (2022-11-01) don't seem address this unless I'm missing something,
Comment 6 Keith Busch 2022-11-09 13:03:49 UTC 
Latest code uses the state machine so that you can't stack resets like this but reported.
Comment 7 John Haxby 2022-11-09 16:22:31 UTC 
Thank you.  I guess this bug can be closed now.
Comment 8 Salvatore Bonaccorso 2022-11-16 13:44:15 UTC 
https://git.kernel.org/linus/1e866afd4bcdd01a70a5eddb4371158d3035ce03 refers to this bugzilla entry.