Massive sending NVME_IOCTL_RESET and NVME_IOCTL_SUBSYS_RESET by user tool at the same time will cause OS panic. Mybe the following modifications in nvme_dev_ioctl can make driver stronger. @@ -3031,15 +3031,23 @@ static long nvme_dev_ioctl(struct file * struct nvme_ctrl *ctrl = file->private_data; void __user *argp = (void __user *)arg; + if (ctrl == NULL) + return -ENOTTY; + if (ctrl->state == NVME_CTRL_DELETING || ctrl->state == NVME_CTRL_DEAD) + return -ENOTTY; + [361245.754472] nvme nvme2: resetting controller [361245.754500] nvme nvme2: resetting controller [361245.761914] nvme nvme2: resetting controller [361245.762081] nvme nvme2: resetting controller [361245.762136] nvme nvme2: resetting controller [361245.762246] Unable to handle kernel paging request at virtual address ffff000081c42020 [361245.762247] Mem abort info: [361245.762248] ESR = 0x96000047 [361245.762250] Exception class = DABT (current EL), IL = 32 bits [361245.762251] SET = 0, FnV = 0 [361245.762251] EA = 0, S1PTW = 0 [361245.762252] Data abort info: [361245.762253] ISV = 0, ISS = 0x00000047 [361245.762254] CM = 0, WnR = 1 [361245.762256] swapper pgtable: 4k pages, 48-bit VAs, pgdp = 00000000c96d3c05 [361245.762258] [ffff000081c42020] pgd=00002027ffffe003, pud=00002027ffffd003, pmd=00000027df6be003, pte=0000000000000000 [361245.762263] Internal error: Oops: 96000047 [#1] SMP [361245.767407] Process trinity-c383 (pid: 2157089, stack limit = 0xffff000099ea8000) [361245.775244] CPU: 95 PID: 2157089 Comm: trinity-c383 Kdump: loaded Tainted: G OE 4.19.90-vhulk2011.1.0.h352.eulerosv2r9.aarch64 #1 [361245.788669] Hardware name: Huawei TaiShan 2280 V2/BC82AMDDA, BIOS 1.08 09/15/2019 [361245.796505] pstate: 60400009 (nZCv daif +PAN -UAO) [361245.801548] pc : nvme_pci_reg_write32+0x30/0x48 [nvme] [361245.806954] lr : nvme_dev_ioctl+0x74/0x1f0 [nvme_core] [361245.812353] sp : ffff000099eabd00 [361245.815856] x29: ffff000099eabd00 x28: ffff80277c503e00 [361245.821434] x27: 0000000000000000 x26: 0000000000000000 [361245.827009] x25: 0000000056000000 x24: ffff8027811fdd68 [361245.832587] x23: ffff8027d26e2700 x22: 0000000000000185 [361245.838164] x21: ffffa0277d04e238 x20: 000000004e564d65 [361245.843740] x19: ffff000081c42020 x18: 0000000000000000 [361245.849315] x17: 0000000000000000 x16: 0000000000000000 [361245.854891] x15: 0000000000000000 x14: 0000000000000000 [361245.860467] x13: 0000000000000000 x12: 0000000000000000 [361245.866043] x11: 0000000000000000 x10: 0000000000000000 [361245.871620] x9 : 0000000000000000 x8 : 0000000000000000 [361245.877197] x7 : ffff000099eabda8 x6 : 0000000000000000 [361245.882774] x5 : 0000000000000000 x4 : 0000000000000000 [361245.888350] x3 : ffff000000b180d0 x2 : 000000004e564d65 [361245.893929] x1 : ffff000081c42000 x0 : ffff000000aaedcc [361245.899508] Call trace: [361245.906638] nvme_pci_reg_write32+0x30/0x48 [nvme] [361245.916366] nvme_dev_ioctl+0x74/0x1f0 [nvme_core] [361245.925984] do_vfs_ioctl+0xc4/0x8c0 [361245.934237] ksys_ioctl+0x8c/0xa0 [361245.942028] __arm64_sys_ioctl+0x28/0x98 [361245.950291] el0_svc_common+0x80/0x1b8 [361245.958202] el0_svc_handler+0x78/0xe0 [361245.965904] el0_svc+0x10/0x260 [361245.972778] Code: d503201f d5033e9f f85682a1 8b334033 (b9000274) [361245.982651] kernel fault(0x1) notification starting on CPU 95 [361245.992186] kernel fault(0x1) notification finished on CPU 95 [361246.001591] [kbox] catch die event on cpu 95. [361246.009623] [kbox] catch die event, start logging. [361246.018015] [kbox] die info:Oops:0047. [361246.025366] [kbox] start to collect. [361246.033241] [kbox] collected_len = 176005, g_log_buf_len_local = 1048576 [361246.043774] [kbox] save kbox pre log success. [361246.052027] [kbox] stack: [361246.058383] [kbox] stack grow form: 0xffff000099eabd00 to 0xffff000099eac000. [361246.073378] [kbox] bd00: ffff000099eabd30 ffff000000aaedcc 0000000000004e45 ffffa0277d04e238 [361246.090743] [kbox] bd20: 0000000000000074 0000000000004e45 ffff000099eabd70 ffff00008038ec2c [361246.109144] [kbox] bd40: ffff00008128a000 0000000000004e45 0000000000000074 ffff00008045ee8c [361246.128431] [kbox] bd60: ffff000081316880 0000000000000074 ffff000099eabe00 ffff00008038f4b4 [361246.148516] [kbox] bd80: ffff8027d26e2700 0000000000000000 ffff8027d26e2700 0000000000000185 [361246.169300] [kbox] bda0: 0000000000004e45 0000000000000074 ffff000099eabd90 ffff0000801f5bc0 [361246.190529] [kbox] bdc0: ffff000099eabde0 2004bb3357eeb400 ffff000099eabe00 ffff00008038f474 [361246.212439] [kbox] bde0: ffff8027d26e2700 ffff000099eabec0 ffff8027d26e2700 2004bb3357eeb400 [361246.235176] [kbox] be00: ffff000099eabe40 ffff00008038f4f0 ffff000099eabec0 ffff000099eabec0 [361246.258513] [kbox] be20: ffffffffffffffff 0000000000000200 ffff000080aa1698 0000000000000015 [361246.282326] [kbox] be40: ffff000099eabe60 ffff000080099e98 000000000000001d ffff000080099e98 [361246.306913] [kbox] be60: ffff000099eabea0 ffff00008009a048 ffff000099eabec0 0000a0275e7d3000 [361246.332091] [kbox] be80: 00000000ffffffff 0000ffff97339614 0000000040000000 ffff000080083c0c [361246.357695] [kbox] bea0: ffff000099eabff0 ffff000080084250 0000000000000200 ffff000080084148
I think what you're showing is the driver accessing a pci register before it has iomapped it. i'll see if there's something we can do about it.
Created attachment 299261 [details] proposed fix Could you confirm if the attached patch fixes your observation?
(In reply to Keith Busch from comment #2) > Created attachment 299261 [details] > proposed fix > > Could you confirm if the attached patch fixes your observation? I've tried this patch. It hung when insmod nvme.ko [Thu Oct 21 15:59:06 2021] INFO: task kworker/u193:2:67453 blocked for more than 120 seconds. [Thu Oct 21 15:59:06 2021] Tainted: G OE 4.19.36-vhulk1907.1.0.h748.eulerosv2r8.aarch64 #1 [Thu Oct 21 15:59:06 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Thu Oct 21 15:59:06 2021] kworker/u193:2 D 0 67453 2 0x00000228 [Thu Oct 21 15:59:06 2021] Workqueue: nvme-reset-wq nvme_reset_work [nvme] [Thu Oct 21 15:59:06 2021] Call trace: [Thu Oct 21 15:59:06 2021] __switch_to+0x94/0xe8 [Thu Oct 21 15:59:06 2021] __schedule+0x31c/0x9c8 [Thu Oct 21 15:59:06 2021] schedule+0x2c/0x88 [Thu Oct 21 15:59:06 2021] schedule_preempt_disabled+0x14/0x20 [Thu Oct 21 15:59:06 2021] __mutex_lock.isra.1+0x1fc/0x540 [Thu Oct 21 15:59:06 2021] __mutex_lock_slowpath+0x24/0x30 [Thu Oct 21 15:59:06 2021] mutex_lock+0x80/0xa8 [Thu Oct 21 15:59:06 2021] nvme_pci_reg_write32+0x3c/0x78 [nvme] [Thu Oct 21 15:59:06 2021] nvme_disable_ctrl+0x40/0x78 [nvme] [Thu Oct 21 15:59:06 2021] nvme_reset_work+0x260/0xea8 [nvme] [Thu Oct 21 15:59:06 2021] process_one_work+0x1b4/0x3f8 [Thu Oct 21 15:59:06 2021] worker_thread+0x210/0x470 [Thu Oct 21 15:59:06 2021] kthread+0x134/0x138 [Thu Oct 21 15:59:06 2021] ret_from_fork+0x10/0x18
Oops, of course it's a circular lock... will rework.
(In reply to Keith Busch from comment #4) > Oops, of course it's a circular lock... will rework. What happened to that? The most recent set of changes (2022-11-01) don't seem address this unless I'm missing something,
Latest code uses the state machine so that you can't stack resets like this but reported.
Thank you. I guess this bug can be closed now.
https://git.kernel.org/linus/1e866afd4bcdd01a70a5eddb4371158d3035ce03 refers to this bugzilla entry.