Bug 212665
Summary: | aoe: kernel crash on blk_update_request: I/O error, BUG: scheduling while atomic | ||
---|---|---|---|
Product: | IO/Storage | Reporter: | Valentin Kleibel (valentin) |
Component: | Other | Assignee: | io_other |
Status: | NEW --- | ||
Severity: | normal | CC: | carnil, valentin |
Priority: | P1 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | 5.10.28 | Subsystem: | |
Regression: | No | Bisected commit-id: | |
Attachments: | patch to fix aoe timeout handling |
Description
Valentin Kleibel
2021-04-13 12:45:53 UTC
Created attachment 300512 [details] patch to fix aoe timeout handling Hello, after lots of tries to tackle the issue and dive into linux block device development, here is a simple procedure for reproducing the issue (no special requirements, just vblade https://github.com/OpenAoE/vblade), a pretty good understanding of the issue and a proposal on how to fix it. *Reproducing the issue:* * prepare "aoe storage backend" (keep vblade running in a seperate shell) ``` apt-get install vblade dd if=/dev/zero of=/root/1g.img bs=1M count=1024 # this creates /dev/etherd/e1.1 served on lo with /root/1g.img as "backend device" vblade 1 1 lo /root/1g.img ``` * afterwards cleanly load aoe.ko and start i/o on the device ``` rmmod aoe # just in case aoe.ko is loaded for some reason #aoe_deadsecs=40 is optional to speed things up, the default timeout is 180s modprobe aoe aoe_iflist=lo aoe_deadsecs=40 dd if=/dev/zero of=/dev/etherd/e1.1 ``` * kill the vblade process * after the timeout the bug is triggered *Understanding the issue* If an aoe device does not respond to any packet on the network layer within aoe_deadsecs, the driver tries to mark the device as down, fail all I/O and clean out the queue. This is done in rexmit_timer() [drivers/block/aoe/aoecmd.c L727ff] which is registered as a timer callback function and never explicitly called. For this reason rexmit_timer() is always called from an interrupt context. When the timeout exceeds aoe_deadsecs aoedev_downdev() is called from rexmit_timer() in L782. aoedev_downdev() then tries to clean out the queue [drivers/block/aoe/aoedev.c L227ff] using blk_mq_freeze_queue() and blk_mq_quiesce_queue(). These funtions will sleep and, in the context of an interrupt, trigger the "BUG: scheduling while atomic" message, ultimately leading to hung tasks and an unusable system. *A proposed fix* Before the change to blk_mq the queue was emptied by iterating over all requests in the queue and failing them 1 by 1 while no new requests were accepted due to ~DEVFL_UP being set. To fail the queue in an atomic-safe way, restoring the previous behavior and adapting it to blk_mq seemed to be the way to go. Fetching requests from the queue is borrowed from nextbuf() [drivers/block/aoe/aoecmd.c L850] and finally failing them is done with aoe_end_request() [drivers/block/aoe/aoecmd.c L1030]. This is done while the ~DEVFL_UP flag is set which prevents aoeblk_queue_rq() [drivers/block/aoe/aoeblk.c L262] to enqueue new requests. Find attached the proposed fix. Cheers, Valentin |