While doing loops of reset_controller and nvme format, mutex timeout/kernel panic occurs Much debug seems to have it failing during nvme_cancel_admin_tagset() I suspect there is a stuck request in a quiesced admin queue Repro: while true; do echo 1 > /sys/class/nvme/nvme0/reset_controller & nvme format -f /dev/nvme0n1 & done
Created attachment 301552 [details] panic log
I'd imagine a fork bomb like this would flood the admin queue with processes wanting to dispatch long running format commands to the point the disabling the controller can't find a tag to delete the IO queues. I guess we could be more preventative of this situation. I'll look into it.
I don't think there's much we can do here. Once you get a few million processes contending for a few dozen tags on an entered request queue, I'm not finding a reasonable to deprioritize the forked formats from starving the reset.
Created attachment 301554 [details] attachment-1781-0.html I don’t think that’s exactly what’s occurring, but I’m not sure. When I was debugging, my results showed a format stuck in idle state because the queue was quiescent and I’m not sure why the wait_completion didn’t finish since it shouldn’t care about idle reqs. ________________________________ From: bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> Sent: Thursday, August 11, 2022 8:07:38 PM To: Jonathan Derrick <Jonathan.Derrick@solidigm.com> Subject: [Bug 216354] nvme: Kernel panic when issueing reset_controller and submitting ioctl [You don't often get email from bugzilla-daemon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Caution: External Email https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216354&data=05%7C01%7Cjonathan.derrick%40solidigm.com%7Cd3dba7d549894d748a1e08da7c077020%7C4bb44aac867a4be0aa3430794e8470dd%7C0%7C0%7C637958668637862319%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CNrYhkv61laQT7zusr3I6Cm%2BbvVnGT69b%2FuFbKnAbkw%3D&reserved=0 --- Comment #3 from Keith Busch (kbusch@kernel.org) --- I don't think there's much we can do here. Once you get a few million processes contending for a few dozen tags on an entered request queue, I'm not finding a reasonable to deprioritize the forked formats from starving the reset. -- You may reply to this email to add a comment. You are receiving this mail because: You reported the bug.
Let me just ask this: are you intentionally running a test that forks as many background formats as possible to run in parallel? Or is that '&' on the format command not intended for this test? When I run this script, the only thing I'm observing is that if the reset aligns just right, the number of format processes that have entered the request_queue will starve the reset_work from deleting the IO queues.
Created attachment 301555 [details] attachment-13103-0.html Well I suppose the intent wasn’t to spawn a million processes, but instead to try and time the race condition. I think it can be reproduced with 50 or 100 iterations. ________________________________ From: bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> Sent: Thursday, August 11, 2022 8:45:10 PM To: Jonathan Derrick <Jonathan.Derrick@solidigm.com> Subject: [Bug 216354] nvme: Kernel panic when issueing reset_controller and submitting ioctl [You don't often get email from bugzilla-daemon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Caution: External Email https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216354&data=05%7C01%7Cjonathan.derrick%40solidigm.com%7C005038a75d2541d6931208da7c0cadaa%7C4bb44aac867a4be0aa3430794e8470dd%7C0%7C0%7C637958691152702989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PQ0bi2cn0Wzs2yYq%2FQKLEd6pm35u%2FjG%2F7pPqzgCFq6s%3D&reserved=0 --- Comment #5 from Keith Busch (kbusch@kernel.org) --- Let me just ask this: are you intentionally running a test that forks as many background formats as possible to run in parallel? Or is that '&' on the format command not intended for this test? When I run this script, the only thing I'm observing is that if the reset aligns just right, the number of format processes that have entered the request_queue will starve the reset_work from deleting the IO queues. -- You may reply to this email to add a comment. You are receiving this mail because: You reported the bug.
I think the number of formats you can spawn is dependent on your h/w. Most will format single threaded, and it will typically take a few seconds to maybe minutes. We have only have 32 admin tags in this driver. If you spawn 100 format dispatching processes, it could be hours before the reset_work can get a tag.
Another possibility here, the passthrough request is trying to flush the scan work before it can free itself, but the scan work can't make progress because the admin queue is quiesced; meanwhile the reset work can't make forward progress because it's waiting for the passthrough request to transition to the idle state.
I think something like that is occurring in the instance I'm reproducing. -----Original Message----- From: bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> Sent: Friday, August 12, 2022 10:17 AM To: Jonathan Derrick <Jonathan.Derrick@solidigm.com> Subject: [Bug 216354] nvme: Kernel panic when issueing reset_controller and submitting ioctl Caution: External Email https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216354&data=05%7C01%7Cjonathan.derrick%40solidigm.com%7Cca43eb7503a341443b9008da7c7e0517%7C4bb44aac867a4be0aa3430794e8470dd%7C0%7C0%7C637959177959797293%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yR6nBe1gIVyZvJX%2FMhhiQiuCCkMLGmxySE79uGP%2FV20%3D&reserved=0 --- Comment #8 from Keith Busch (kbusch@kernel.org) --- Another possibility here, the passthrough request is trying to flush the scan work before it can free itself, but the scan work can't make progress because the admin queue is quiesced; meanwhile the reset work can't make forward progress because it's waiting for the passthrough request to transition to the idle state. -- You may reply to this email to add a comment. You are receiving this mail because: You reported the bug.