Bug 211711
| Summary: | Kernel panic when qemu with vhost-vdpa net device is started the second time | ||
|---|---|---|---|
| Product: | Drivers | Reporter: | Gautam Dawar (gdawar) |
| Component: | Network | Assignee: | drivers_network (drivers_network) |
| Status: | RESOLVED CODE_FIX | ||
| Severity: | high | CC: | gdawar, jasowang, m.s.tsirkin |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 5.9 and above | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
crash log
Patch on kernel v5.9.0 Patch v2 on kernel v5.9.0 Patch v2 on kernel v5.9.0 |
||
Root cause analysis:
====================
When the VM is powered off, vhost_dev_stop() is invoked which in turn calls vhost_vdpa_reset_device() causing the irq_bypass producers to be un-registered.
On the next run, when qemu opens the vhost device, the vhost_vdpa_open() file operation calls vhost_dev_init(). Here, call_ctx->producer memory is cleared in vhost_vring_call_reset().
Further, when the virtqueues are initialized by vhost_virtqueue_init(), vhost_vdpa_setup_vq_irq() again registers the irq_bypass producer for each virtqueue. As the node member of struct irq_bypass_producer is also initialized to zero, traversal on the producers list causes crash due to NULL pointer dereference.
Fix details:
=============
I think that this issue can be fixed by invoking vhost_vdpa_setup_vq_irq() only when vhost_vdpa_set_status() includes VIRTIO_CONFIG_S_DRIVER_OK in the new status value. This way, there won’t be any stale nodes in the irqbypass module’s producers list which are reset in vhost_vring_call_reset().
Patch:
======
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 62a9bb0efc55..fdad94e2fbf9 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -409,7 +409,6 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
cb.private = NULL;
}
ops->set_vq_cb(vdpa, idx, &cb);
- vhost_vdpa_setup_vq_irq(v, idx);
break;
case VHOST_SET_VRING_NUM:
Created attachment 295241 [details]
Patch on kernel v5.9.0
When the VM is powered off, vhost_vdpa_clean_irq() misses on calling
irq_bypass_unregister_producer() for irq 0 because of existing check.
This leaves stale producer nodes, which are reset in vhost_vring_call_reset()
when vhost_dev_init() is invoked during second qemu run.
As the node member of struct irq_bypass_producer is also initialized
to zero, traversal on the producers list causes crash due to NULL pointer
dereference.
Here is the patch to fix this issue:
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 62a9bb0efc55..d1c3a33c6239 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -849,7 +849,7 @@ static void vhost_vdpa_clean_irq(struct vhost_vdpa *v)
for (i = 0; i < v->nvqs; i++) {
vq = &v->vqs[i];
- if (vq->call_ctx.producer.irq)
+ if (vq->call_ctx.producer.irq >= 0)
irq_bypass_unregister_producer(&vq->call_ctx.producer);
}
}
Created attachment 295349 [details]
Patch v2 on kernel v5.9.0
Created attachment 295351 [details]
Patch v2 on kernel v5.9.0
-----
changelog:
v1->v2:
- Addressed Jason's comment to remove the irq check and use
vhost_vdpa_unsetup_vq_irq() to avoid local variable vq
-----
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 62a9bb0efc55..e00573b87aba 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -844,14 +844,10 @@ static int vhost_vdpa_open(struct inode *inode, struct file *filep)
static void vhost_vdpa_clean_irq(struct vhost_vdpa *v)
{
- struct vhost_virtqueue *vq;
int i;
- for (i = 0; i < v->nvqs; i++) {
- vq = &v->vqs[i];
- if (vq->call_ctx.producer.irq)
- irq_bypass_unregister_producer(&vq->call_ctx.producer);
- }
+ for (i = 0; i < v->nvqs; i++)
+ vhost_vdpa_unsetup_vq_irq(v, i);
}
static int vhost_vdpa_release(struct inode *inode, struct file *filep)
The patch has been upstreamed to 5.10-stable & 5.11-stable trees which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
vhost_vdpa-fix-the-missing-irq_bypass_unregister_producer-invocation.patch
and it can be found in the queue-5.10 and queue-5.11 subdirectories.
[PATCH 5.10 047/157] vhost_vdpa: fix the missing irq_bypass_unregister_producer() invocation
commit 4c050286bb202cffd5467c1cba982dff391d62e1 upstream.
[PATCH 5.11 052/120] vhost_vdpa: fix the missing irq_bypass_unregister_producer() invocation
commit 4c050286bb202cffd5467c1cba982dff391d62e1 upstream.
|
Created attachment 295239 [details] crash log When qemu with vhost-vdpa netdevice is run for the first time, it works well but after the VM is powered off (or qemu is killed), on starting qemu again causes kernel panic due to NULL pointer dereference in irq_bypass_register_producer. The issue is not seen if the .get_vq_irq callback in vdpa_config_ops is not implemented (set to NULL) as in that case vhost_vdpa_setup_vq_irq() and hence irq_bypass_register_producer() is not invoked which is the area of crash. Here is the complete command for qemu invocation: qemu-system-x86_64 -machine accel=kvm -m 2G -hda /dawarg/centos82.qcow2 -name gautam,process=gautamd -enable-kvm -netdev vhost-vdpa,id=mynet0,vhostdev=/dev/vhost-vdpa-0 -device virtio-net-pci,netdev=mynet0,mac=02:AA:BB:DD:00:20,disable-modern=off,page-per-vq=on -cpu host --nographic