Bug 15833 - NETDEV WATCHDOG: ethX (e1000e): transmit queue 0 timed out
Summary: NETDEV WATCHDOG: ethX (e1000e): transmit queue 0 timed out
Status: RESOLVED OBSOLETE
Alias: None
Product: Other
Classification: Unclassified
Component: Bug Tracker (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Bug Me Administrator
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-04-22 10:18 UTC by ZS1982
Modified: 2014-01-05 15:35 UTC (History)
3 users (show)

See Also:
Kernel Version: 2.6.33.2
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description ZS1982 2010-04-22 10:18:00 UTC
I have a card inter-82583v with 4 port, and do some storm-testing in bridge mode:

br0             8000.080027f6c278       no              eth6 eth7 eth8 eth9


After a while, i found that there're two port up/down continuous after a message:"NETDEV_WATCHDOG:....".


please see the attatched file for detail.
thanks.
Comment 1 Richard Wall 2010-04-26 09:52:06 UTC
I'm seeing a similar problem. We have a pair of very busy Squid servers (peak ~15000 TCP connections, ~100Mb/sec) using identical Asus server motherboards. The networking on one box intermittently fails - sometimes after a few hours sometimes after as much as 7 days.

Here's the relevant info and an extract from syslog / dmesg

# uname -r
2.6.31.12

# cat /sys/class/net/eth0/device/vendor 
0x14e4
# cat /sys/class/net/eth0/device/device 
0x1659

{{{
http://www.pcidatabase.com/vendor_details.php?id=767
0x1659	
Chip Number:	BCM5721
Chip Description:	NetXtreme Gigabit Ethernet PCI Express
}}}

# ethtool -i eth0
driver: tg3
version: 3.99
firmware-version: 5721-v3.65
bus-info: 0000:03:00.0

#cat /var/log/syslog
Apr 22 23:49:35  kernel: [623449.988504] ------------[ cut here ]------------
Apr 22 23:49:35  kernel: [623449.988511] WARNING: at net/sched/sch_generic.c:246 dev_watchdog+0x1be/0x1d0()
Apr 22 23:49:35  kernel: [623449.988514] Hardware name: System Product Name
Apr 22 23:49:35  kernel: [623449.988516] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Apr 22 23:49:35  kernel: [623449.988518] Modules linked in: ip_gre e1000 via_rhine tg3 libphy r8169 pcnet32 e100 8139too mii w83627ehf vt8231 via686a hwm
on_vid coretemp asus_atk0110 hwmon
Apr 22 23:49:35  kernel: [623449.988533] Pid: 0, comm: swapper Not tainted 2.6.31.12 #2
Apr 22 23:49:35  kernel: [623449.988535] Call Trace:
Apr 22 23:49:35  kernel: [623449.988540]  [<c012864e>] ? warn_slowpath_common+0x6e/0xb0
Apr 22 23:49:35  kernel: [623449.988543]  [<c034e4fe>] ? dev_watchdog+0x1be/0x1d0
Apr 22 23:49:35  kernel: [623449.988546]  [<c01286db>] ? warn_slowpath_fmt+0x2b/0x30
Apr 22 23:49:35  kernel: [623449.988549]  [<c034e4fe>] ? dev_watchdog+0x1be/0x1d0
Apr 22 23:49:35  kernel: [623449.988553]  [<c011f842>] ? __wake_up+0x42/0x60
Apr 22 23:49:35  kernel: [623449.988557]  [<c01376f2>] ? insert_work+0x42/0x50
Apr 22 23:49:35  kernel: [623449.988560]  [<c034e340>] ? dev_watchdog+0x0/0x1d0
Apr 22 23:49:35  kernel: [623449.988564]  [<c0131149>] ? run_timer_softirq+0xf9/0x1c0
Apr 22 23:49:35  kernel: [623449.988567]  [<c012d2c0>] ? __do_softirq+0x80/0x100
Apr 22 23:49:35  kernel: [623449.988570]  [<c012d36d>] ? do_softirq+0x2d/0x40
Apr 22 23:49:35  kernel: [623449.988574]  [<c0114d94>] ? smp_apic_timer_interrupt+0x54/0x90
Apr 22 23:49:35  kernel: [623449.988577]  [<c0103676>] ? apic_timer_interrupt+0x2a/0x30
Apr 22 23:49:35  kernel: [623449.988581]  [<c03f00d8>] ? klist_add_before+0x18/0x50
Apr 22 23:49:35  kernel: [623449.988585]  [<c0109dc2>] ? mwait_idle+0x42/0x60
Apr 22 23:49:35  kernel: [623449.988587]  [<c0101d55>] ? cpu_idle+0x35/0x60
Apr 22 23:49:35  kernel: [623449.988590] ---[ end trace 346a74434bf31555 ]---
Apr 22 23:49:35  kernel: [623449.988592] tg3: eth0: transmit timed out, resetting
Apr 22 23:49:35  kernel: [623449.988596] tg3: DEBUG: MAC_TX_STATUS[0000000f] MAC_RX_STATUS[00000008]
Apr 22 23:49:35  kernel: [623449.988601] tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Apr 22 23:49:35  kernel: [623450.089694] tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
Apr 22 23:49:35  kernel: [623450.248154] tg3: eth0: Link is down.
Comment 2 Alan 2012-11-20 17:20:34 UTC
Closing as obsolete, if this is still seen with modern kernels please re-open and update
Comment 3 Jo Mills 2014-01-05 15:35:24 UTC
(In reply to Alan from comment #2)
> Closing as obsolete, if this is still seen with modern kernels please
> re-open and update

Please excuse me if I am incorrectly updating this bug report. The symptoms I am seeing are very similar to those shown in Comment 1, but for a via-rhine card and with a relatively new kernel.  The problem happens after I try and make eth1 use the xen-pciback driver and then create a VM to make use of this NIC. (I have two servers connected via DRBD link, the VM runs just fine on my yet to be upgraded "squeeze" server).  The firmware on the "wheezy" server has been upgraded to the latest version (Intel S3200SHV motherboard).



# xm dmesg
(XEN) Xen version 4.1.4 (Debian 4.1.4-3+deb7u1) (carnil@debian.org) (gcc version 4.7.2 (Debian 4.7.2-5) ) Sun May  5 14:44:49 UTC 2013
(XEN) Bootloader: GRUB 1.99-27+deb7u2
(XEN) Command line: placeholder dom0_mem=2G,max:2G dom0_max_vcpus=1 dom0_vcpus_pin pcie_aspm=off
   .
   .
   .


# uname -r
3.2.0-4-amd64

# cat /sys/class/net/eth0/device/vendor
0x1106

# cat /sys/class/net/eth0/device/device
0x3106

http://www.pcidatabase.com/vendor_details.php?id=648
0x3106      Chip Number: 	VT6105M/LOM
            Chip Description: 	Rhine III PCI Fast Ethernet Controller


# ethtool -i eth0
version: 1.5.0
firmware-version: 
bus-info: 0000:03:00.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no


# lspci -k
    .
    .
    .
03:00.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
        Subsystem: D-Link System Inc DFE-530TX rev C
        Kernel driver in use: via-rhine
03:01.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
        Subsystem: D-Link System Inc DFE-530TX rev C
        Kernel driver in use: via-rhine
03:02.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
        Subsystem: Intel Corporation Device 34d0
        Kernel driver in use: e1000


Jan  5 14:56:40 green kernel: [  454.820064] ------------[ cut here ]------------
Jan  5 14:56:40 green kernel: [  454.820073] WARNING: at /build/linux-rrsxby/linux-3.2.51/net/sched/sch_generic.c:256 dev_watchdog+0xf2/0x151()
Jan  5 14:56:40 green kernel: [  454.820075] Hardware name: S3210SH
Jan  5 14:56:40 green kernel: [  454.820076] NETDEV WATCHDOG: eth0 (via-rhine): transmit queue 0 timed out
Jan  5 14:56:40 green kernel: [  454.820078] Modules linked in: xen_blkback xen_blkfront xen_gntdev xen_evtchn xenfs drbd lru_cache bridge stp ext2 xen_pciback loop snd_pcm snd_page_alloc snd_timer snd soundcore psmouse serio_raw coretemp i2c_i801 iTCO_wdt pcspkr iTCO_vendor_support i2c_core evdev acpi_cpufreq mperf i3200_edac processor button thermal_sys edac_core ext3 mbcache jbd dm_mod sg sd_mod sr_mod crc_t10dif cdrom floppy via_rhine ata_generic e1000 mii ata_piix libata uhci_hcd ehci_hcd scsi_mod usbcore usb_common [last unloaded: scsi_wait_scan]
Jan  5 14:56:40 green kernel: [  454.820110] Pid: 0, comm: swapper/0 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.51-1
Jan  5 14:56:40 green kernel: [  454.820112] Call Trace:
Jan  5 14:56:40 green kernel: [  454.820114]  <IRQ>  [<ffffffff81046cbd>] ? warn_slowpath_common+0x78/0x8c
Jan  5 14:56:40 green kernel: [  454.820121]  [<ffffffff81046d69>] ? warn_slowpath_fmt+0x45/0x4a
Jan  5 14:56:40 green kernel: [  454.820123]  [<ffffffff812a6f11>] ? netif_tx_lock+0x40/0x75
Jan  5 14:56:40 green kernel: [  454.820127]  [<ffffffff812a7081>] ? dev_watchdog+0xf2/0x151
Jan  5 14:56:40 green kernel: [  454.820130]  [<ffffffff81052480>] ? run_timer_softirq+0x19a/0x261
Jan  5 14:56:40 green kernel: [  454.820133]  [<ffffffff8109102c>] ? handle_irq_event_percpu+0x15f/0x17d
Jan  5 14:56:40 green kernel: [  454.820135]  [<ffffffff812a6f8f>] ? netif_tx_unlock+0x49/0x49
Jan  5 14:56:40 green kernel: [  454.820138]  [<ffffffff8104c2f8>] ? __do_softirq+0xb9/0x177
Jan  5 14:56:40 green kernel: [  454.820142]  [<ffffffff8121bb21>] ? __xen_evtchn_do_upcall+0x24a/0x287
Jan  5 14:56:40 green kernel: [  454.820145]  [<ffffffff8135646c>] ? call_softirq+0x1c/0x30
Jan  5 14:56:40 green kernel: [  454.820148]  [<ffffffff8100f8cd>] ? do_softirq+0x3c/0x7b
Jan  5 14:56:40 green kernel: [  454.820150]  [<ffffffff8104c560>] ? irq_exit+0x3c/0x99
Jan  5 14:56:40 green kernel: [  454.820153]  [<ffffffff8121cee1>] ? xen_evtchn_do_upcall+0x27/0x32
Jan  5 14:56:40 green kernel: [  454.820155]  [<ffffffff813564be>] ? xen_do_hypervisor_callback+0x1e/0x30
Jan  5 14:56:40 green kernel: [  454.820157]  <EOI>  [<ffffffff810013aa>] ? hypercall_page+0x3aa/0x1000
Jan  5 14:56:40 green kernel: [  454.820161]  [<ffffffff810013aa>] ? hypercall_page+0x3aa/0x1000
Jan  5 14:56:40 green kernel: [  454.820164]  [<ffffffff8100675a>] ? xen_safe_halt+0xc/0x13
Jan  5 14:56:40 green kernel: [  454.820167]  [<ffffffff810144c0>] ? default_idle+0x47/0x7f
Jan  5 14:56:40 green kernel: [  454.820169]  [<ffffffff8100d24d>] ? cpu_idle+0xaf/0xf2
Jan  5 14:56:40 green kernel: [  454.820172]  [<ffffffff816abb36>] ? start_kernel+0x3b8/0x3c3
Jan  5 14:56:40 green kernel: [  454.820175]  [<ffffffff816ad4da>] ? xen_start_kernel+0x412/0x418
Jan  5 14:56:40 green kernel: [  454.820176] ---[ end trace 29390ca02b709849 ]---

Note You need to log in before you can comment on or make changes to this bug.