Bug 213179
Summary: | vdpa_device_unregister() caller hangs if the QEMU (VM) using the vhost_vdpa device is running | ||
---|---|---|---|
Product: | Drivers | Reporter: | Gautam Dawar (gdawar) |
Component: | Network | Assignee: | drivers_network (drivers_network) |
Status: | NEW --- | ||
Severity: | high | ||
Priority: | P1 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | 5.11 and above | Subsystem: | |
Regression: | No | Bisected commit-id: |
Description
Gautam Dawar
2021-05-22 20:04:29 UTC
Here is the sequence of events that cause the rmmod sfc command to hang: 1. QEMU opens the vhost-vdpa character device for configuring the underlying vDPA device. At this time vhost_vdpa_open callback of vhost-vdpa kernel module is invoked. This function sets the opened atomic variable to 1. opened = atomic_cmpxchg(&v->opened, 0, 1); 2. This vdpa character device is generally closed when the VM is shutdown. At this time (first and last close() of the vhost-vdpa char device fd), vhost_vdpa module's release callback vhost_vdpa_release gets called which decrements the opened variable and issues the completion event: atomic_dec(&v->opened); complete(&v->completion); 3. When sfc is unloaded, vdpa_device_unregister is invoked for each vdpa device which in-turn invokes vhost_vdpa_remove in vhost-vdpa module. vhost-vdpa kernel module waits (wait_for_completion) on the atomic variable 'opened' which refers to the state of vdpa character device. Since the VM was still running when sfc unload is attempted, the character device remains open and vhost_vdpa module's vhost_vdpa_release is not called. Because of this, vhost_vdpa_remove blocks on the completion event for hung_task_timeout_secs which is 120 seconds. To fix this problem, vhost-vdpa module must notify qemu to close the fd corresponding to the vhost-vdpa character device when it is waiting for the completion event in vhost_vdpa_remove() In continuation to my findings noted in bug description and previous comment, I found that any malicious user-space application can render a module registering a vDPA device to hang in their de-initialization sequence. This will typically surface when vdpa_device_unregister() is called from the function responsible for module unload leading rmmod commands to not return, forever. This is being discussed at https://lists.linuxfoundation.org/pipermail/virtualization/2021-June/054775.html |