Bug 16413 - Kernel BUG and guest hang on network activity
Summary: Kernel BUG and guest hang on network activity
Status: RESOLVED CODE_FIX
Alias: None
Product: Networking
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Michael S. Tsirkin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-07-19 00:25 UTC by Alex Unigovsky
Modified: 2010-07-23 15:35 UTC (History)
5 users (show)

See Also:
Kernel Version: 2.6.34.1 (gentoo 2.6.34-r2)
Subsystem:
Regression: No
Bisected commit-id:


Attachments
Full dmesg from host machine (86.50 KB, text/plain)
2010-07-19 00:25 UTC, Alex Unigovsky
Details
lspci -vv from host machine (75.26 KB, text/plain)
2010-07-19 00:30 UTC, Alex Unigovsky
Details
Kernel .config from host machine (91.52 KB, text/plain)
2010-07-19 00:35 UTC, Alex Unigovsky
Details
Kernel .config with NETPOLL disabled (65.47 KB, text/plain)
2010-07-19 11:45 UTC, Alex Unigovsky
Details
debugging patch for tun (840 bytes, application/octet-stream)
2010-07-19 13:32 UTC, Michael S. Tsirkin
Details
Full dmesg with diagnostic patch applied (67.04 KB, text/plain)
2010-07-19 17:50 UTC, Alex Unigovsky
Details
gro: Fix bogus gso_size on the first fraglist entry (1.09 KB, application/octet-stream)
2010-07-19 23:56 UTC, Herbert Xu
Details

Description Alex Unigovsky 2010-07-19 00:25:45 UTC
Created attachment 27138 [details]
Full dmesg from host machine

Kernel produces BUG(), hangs qemu process and generally misbehaves on inbound scp traffic. Further attempts to connect to libvirtd fail.

After SIGKILL'ing hanging qemu and libvirtd processes, cleaning stale PID-files etc. libvirt and its guests are able to start again. But they use new vnetX interfaces. Old vnetX interfaces just sit there. After downing them with ifconfig, an attempt to delete these interfaces with command 'ip link delete vnet0' hangs with kernel spamming 'kernel:unregister_netdevice: waiting for vnet0 to become free. Usage count = 1' to syslog.

This BUG() and hanging is reproducible 100% time.

Host configuration
------------------

x86_64, Gentoo
16-way (2x Xeon X5550)
Kernel 2.6.34.1 (gentoo 2.6.34-r2)
Partial list of interfaces:

 * eth0
   192.168.0.1/24
   SCP connection came from here.

 * br0
   188.114.196.59/29
   Guests join this bridge.
   Enslaved physical interfaces: vlan108, vnetX.

 * vlan108
   Sits on top of eth1, using 802.1Q tagging.
   No IP address.
   Part of br0 bridge.

 * eth1
   No IP address.

vhost_net loaded and used by TUN/TAP.
lspci -vv and full dmesg are attached.
Libvirt listens on eth0. Kerberos is used for auth.

Guest
-----

Fedora 13 x86_64
Uses 2 cores.
Uses 1536MB RAM.
32GB LVM volume as the only block device (provided by host).
Comment 1 Alex Unigovsky 2010-07-19 00:30:58 UTC
Created attachment 27139 [details]
lspci -vv from host machine
Comment 2 Alex Unigovsky 2010-07-19 00:35:45 UTC
Created attachment 27140 [details]
Kernel .config from host machine
Comment 3 Michael S. Tsirkin 2010-07-19 10:21:12 UTC
Looks like a bug in NETPOLL in host bridge code.
please try disabling CONFIG_NET_POLL_CONTROLLER
(and/or all of netpoll) and try to reproduce this.

Thanks!
Comment 4 Alex Unigovsky 2010-07-19 11:41:03 UTC
Nope, no luck. I'm reproducing this without NETPOLL:

aegis ~ # zcat /proc/config.gz |grep NET_POLL
# CONFIG_NET_POLL_CONTROLLER is not set
aegis ~ # zcat /proc/config.gz |grep NETPOLL
# CONFIG_NETPOLL is not set

I'm attaching new dmesg.

Further details:
SCP connection attempt is made from a host in 192.168.2.0/24 subnet, which is
routed on eth0 via OSPF (quagga). Initial SSH auth succeeds, and guest always
hangs only after first file of the batch (scp) is transferred.

That connection request passes NAT inside this box.
Also this box uses multiple routing tables (ip route ... table X, ip rule), in
case this matters.
Comment 5 Alex Unigovsky 2010-07-19 11:45:42 UTC
Created attachment 27149 [details]
 Kernel .config with NETPOLL disabled

aegis ~ # zcat /proc/config.gz |grep NET_POLL
# CONFIG_NET_POLL_CONTROLLER is not set
aegis ~ # zcat /proc/config.gz |grep NETPOLL
# CONFIG_NETPOLL is not set
Comment 6 Alex Unigovsky 2010-07-19 11:51:34 UTC
Bah, sorry. Previous attachment was dmesg, not config.
Comment 7 Michael S. Tsirkin 2010-07-19 12:07:50 UTC
So if I understand it correctly, we have the
packet with a wrong gso_type come in
through igb_poll?
Comment 8 Michael S. Tsirkin 2010-07-19 12:10:19 UTC
sorry assigned by mistake.
Comment 9 Herbert Xu 2010-07-19 12:12:27 UTC
I see no definitive proof that it is coming from igb_poll.

I think what would help is to print out some more info about the bogus packet.  We should probably make that permanent as this isn't the first time we've had a crash at this spot and had to hunt for more information.
Comment 10 Michael S. Tsirkin 2010-07-19 13:32:02 UTC
Created attachment 27152 [details]
debugging patch for tun

pls try this patch (might need minor tweaks to appy to gentoo kernel)
this replaces BUG with WARN and adds extra printouts
Comment 11 Alex Unigovsky 2010-07-19 17:50:09 UTC
Created attachment 27156 [details]
Full dmesg with diagnostic patch applied

With this no processes/guests hang, and traffic is undisturbed, but still triggers WARNs.

Hoping that the attached log will help squashing the bug. Thanks.
Comment 12 Alexander Duyck 2010-07-19 18:03:00 UTC
My guess would be that this is due to some sort of interaction with GRO since the in-kernel igb driver doesn't support LRO and does not set gso_size or gso_type.

One thing you might try would be to disable GRO via "ethtool -K <ethX> gro off".  If that resolves the issue it is likely something in the way that igb is working with GRO to aggregate packets.
Comment 13 Alex Unigovsky 2010-07-19 18:11:56 UTC
(In reply to comment #12)
> My guess would be that this is due to some sort of interaction with GRO since
> the in-kernel igb driver doesn't support LRO and does not set gso_size or
> gso_type.
> 
> One thing you might try would be to disable GRO via "ethtool -K <ethX> gro
> off".  If that resolves the issue it is likely something in the way that igb
> is
> working with GRO to aggregate packets.

Yes, indeed, switching GRO off on eth0 (incoming interface) made all the WARNs disappear.
Comment 14 Herbert Xu 2010-07-19 23:56:11 UTC
Created attachment 27161 [details]
gro: Fix bogus gso_size on the first fraglist entry

Does this patch help?
Comment 15 Alex Unigovsky 2010-07-20 07:00:28 UTC
(In reply to comment #14)
> gro: Fix bogus gso_size on the first fraglist entry
> 
> Does this patch help?

Yes, it does. Now I can no longer get WARNs even with GRO enabled. Thank you very much.
Comment 16 Alex Unigovsky 2010-07-23 15:35:41 UTC
I'm closing the bug. Is it ok?

Note You need to log in before you can comment on or make changes to this bug.