12877 – tg3: eth0 transit timed out, resetting -> dead NIC

Bug 12877 - tg3: eth0 transit timed out, resetting -> dead NIC

Summary: tg3: eth0 transit timed out, resetting -> dead NIC

Status:	REOPENED

Alias:	None

Product:	Drivers
Classification:	Unclassified
Component:	Network (show other bugs)
Hardware:	All Linux

Importance:	P1 high
Assignee:	drivers_network@kernel-bugs.osdl.org

URL:
Keywords:

Duplicates (1):	205717 (view as bug list)
Depends on:
Blocks:

Reported:	2009-03-15 07:22 UTC by Bernhard Schmidt
Modified:	2024-04-17 02:14 UTC (History)
CC List:	34 users (show)

See Also:
Kernel Version:	3.13
Subsystem:
Regression:	No
Bisected commit-id:

Attachments
transmit timed out - Vanilla 5.3.11 with txo and gxo off (85.91 KB, text/plain) 2019-11-23 07:05 UTC, Nathan Shearer	Details
mac-mini-6,2-C07JG6DZDWYN-dmidecode (28.17 KB, text/plain) 2020-09-27 04:44 UTC, Nathan Shearer	Details
linux-4.4.203-config-working (183.67 KB, text/plain) 2020-09-27 04:45 UTC, Nathan Shearer	Details
linux-5.4-config-affected (234.57 KB, text/plain) 2020-09-27 04:45 UTC, Nathan Shearer	Details
tg3_timing_out_akosiaris.dmesg (37.10 KB, text/plain) 2021-11-01 17:06 UTC, Alexandros Kosiaris	Details
attachment-12320-0.html (2.66 KB, text/html) 2022-01-01 04:56 UTC, Andrew	Details
Add an attachment (proposed patch, testcase, etc.)

Description Bernhard Schmidt 2009-03-15 07:22:58 UTC

Latest working kernel version: none
Earliest failing kernel version: 2.6.28.1
Distribution: Debian Lenny
Hardware Environment: HP DL320G5p
Software Environment: Debian Lenny host for KVM VMs
Problem Description:

Every couple of weeks the network of my colo box dies with the following message:

[784060.816020] ------------[ cut here ]------------
[784060.869153] WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x121/0x1b8()
[784060.953146] NETDEV WATCHDOG: eth0 (tg3): transmit timed out
[784061.018138] Modules linked in: esp6 xfrm6_mode_tunnel authenc esp4 xfrm4_mode_tunnel tun kvm_intel kvm xt_NOTRACK ip6table_raw ip6t_LOG nf_conntrack_ipv6 ip6table_filter ip6_tables xt_physdev ipt_LOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_hashlimit iptable_filter ip_tables x_tables bridge stp llc deflate zlib_deflate zlib_inflate ctr twofish twofish_common camellia serpent blowfish des_generic cbc aes_x86_64 aes_generic xcbc sha256_generic sha1_generic crypto_null af_key dm_crypt ipv6 coretemp loop ipmi_si ipmi_msghandler hpilo hpwdt pcspkr shpchp pci_hotplug container button psmouse serio_raw evdev ext3 jbd dm_mirror dm_region_hash dm_log dm_snapshot dm_mod raid1 md_mod sg sd_mod sr_mod cdrom usbhid hid ata_piix ata_generic libata scsi_mod ide_pci_generic ide_core ehci_hcd tg3 libphy uhci_hcd thermal processor fan thermal_sys
[784061.891133] Pid: 0, comm: swapper Not tainted 2.6.28.7 #1
[784061.954129] Call Trace:
[784061.983133]  <IRQ>  [<ffffffff802398aa>] warn_slowpath+0xb4/0xda
[784062.053147]  [<ffffffffa0269998>] dst_output+0x0/0xb [ipv6]
[784062.118130]  [<ffffffff803e35a8>] nf_hook_slow+0x62/0xc3
[784062.180139]  [<ffffffffa0269998>] dst_output+0x0/0xb [ipv6]
[784062.245126]  [<ffffffff80332437>] __next_cpu+0x19/0x26
[784062.305124]  [<ffffffff80212a61>] read_tsc+0xa/0x1f
[784062.362126]  [<ffffffff80251c69>] getnstimeofday+0x52/0xac
[784062.426126]  [<ffffffff803d9bd1>] dev_watchdog+0x121/0x1b8
[784062.490124]  [<ffffffff8025015f>] sched_clock_tick+0x8a/0x92
[784062.556124]  [<ffffffff803d9ab0>] dev_watchdog+0x0/0x1b8
[784062.618123]  [<ffffffff80241f50>] run_timer_softirq+0x198/0x21a
[784062.687118]  [<ffffffff80251c69>] getnstimeofday+0x52/0xac
[784062.751117]  [<ffffffff8023e61a>] __do_softirq+0x83/0x143
[784062.814116]  [<ffffffff8020d6ec>] call_softirq+0x1c/0x28
[784062.876119]  [<ffffffff8020ecd0>] do_softirq+0x3c/0x81
[784062.936114]  [<ffffffff8023e338>] irq_exit+0x3f/0x83
[784062.994139]  [<ffffffff8021be99>] smp_apic_timer_interrupt+0x92/0xab
[784063.068116]  [<ffffffff8020cef8>] apic_timer_interrupt+0x88/0x90
[784063.138109]  <EOI>  [<ffffffffa03c4a20>] handle_halt+0x0/0x12 [kvm_intel]
[784063.217116]  [<ffffffff802134fe>] mwait_idle+0x3c/0x46
[784063.277113]  [<ffffffff8020b0bd>] cpu_idle+0x51/0x92
[784063.335127] ---[ end trace 444b547394c96982 ]---
[784063.389142] tg3: eth0: transmit timed out, resetting
[784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff] MAC_RX_STATUS[ffffffff]
[784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]
[784063.706035] tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
[784063.875340] tg3: tg3_stop_block timed out, ofs=2000 enable_bit=2
[784064.044372] tg3: tg3_stop_block timed out, ofs=2400 enable_bit=2
[784064.213191] tg3: tg3_stop_block timed out, ofs=2800 enable_bit=2
[784064.382454] tg3: tg3_stop_block timed out, ofs=3000 enable_bit=2
[784064.551295] tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
[784064.720269] tg3: tg3_stop_block timed out, ofs=1800 enable_bit=2
[784064.889183] tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
[784065.057321] tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
[784065.226318] tg3: tg3_stop_block timed out, ofs=1000 enable_bit=2
[784065.395423] tg3: tg3_stop_block timed out, ofs=1c00 enable_bit=2
[784065.564199] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
[784065.769278] tg3: tg3_stop_block timed out, ofs=3c00 enable_bit=2
[784065.938319] tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
[784067.283239] tg3: eth0: No firmware running.
[784068.533652] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
[784081.605984] tg3: eth0: Link is down.

When it happens I either have to reboot the system or rmmod/modprobe tg3 to get it working again. The interface affected is the routed upstream port of the system, the system doesn't do much more than to route/firewall to an internal bridge where several KVM VMs are connected to. eth0 has a shared physical port with the on-board iLO2, which is still reachable when the problem happens. The switchport bounces a couple of times though.

Steps to reproduce:

Comment 1 Bernhard Schmidt 2009-03-15 09:00:52 UTC

svr02:~# lspci -v
00:00.0 Host bridge: Intel Corporation 3200/3210 Chipset DRAM Controller (rev 01)
        Subsystem: Hewlett-Packard Company Device 31f9                           
        Flags: bus master, fast devsel, latency 0                                
        Capabilities: [e0] Vendor Specific Information <?>                       

00:01.0 PCI bridge: Intel Corporation 3200/3210 Chipset Host-Primary PCI Express Bridge (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                            
        Bus: primary=00, secondary=15, subordinate=1a, sec-latency=0                                                         
        Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9                                                    
        Capabilities: [80] Power Management version 3                                                                        
        Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                                      
        Capabilities: [a0] Express Root Port (Slot+), MSI 00                                                                 
        Capabilities: [100] Virtual Channel <?>                                                                              
        Capabilities: [140] Root Complex Link <?>                                                                            
        Kernel driver in use: pcieport-driver                                                                                
        Kernel modules: shpchp                                                                                               

00:06.0 PCI bridge: Intel Corporation 3210 Chipset Host-Secondary PCI Express Bridge (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                         
        Bus: primary=00, secondary=12, subordinate=14, sec-latency=0                                                      
        Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9                                                 
        Capabilities: [80] Power Management version 3                                                                     
        Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                                   
        Capabilities: [a0] Express Root Port (Slot+), MSI 00                                                              
        Capabilities: [100] Virtual Channel <?>                                                                           
        Capabilities: [140] Root Complex Link <?>                                                                         
        Kernel driver in use: pcieport-driver                                                                             
        Kernel modules: shpchp                                                                                            

00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                  
        Bus: primary=00, secondary=02, subordinate=03, sec-latency=0                                               
        Memory behind bridge: fdf00000-fdffffff                                                                    
        Capabilities: [40] Express Root Port (Slot-), MSI 00                                                       
        Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                            
        Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000                                                
        Capabilities: [a0] Power Management version 2                                                              
        Capabilities: [100] Virtual Channel <?>                                                                    
        Capabilities: [180] Root Complex Link <?>                                                                  
        Kernel driver in use: pcieport-driver                                                                      
        Kernel modules: shpchp                                                                                     

00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                  
        Bus: primary=00, secondary=04, subordinate=06, sec-latency=0                                               
        Capabilities: [40] Express Root Port (Slot-), MSI 00                                                       
        Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                            
        Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000                                                
        Capabilities: [a0] Power Management version 2                                                              
        Capabilities: [100] Virtual Channel <?>                                                                    
        Capabilities: [180] Root Complex Link <?>                                                                  
        Kernel driver in use: pcieport-driver                                                                      
        Kernel modules: shpchp                                                                                     

00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                  
        Bus: primary=00, secondary=07, subordinate=09, sec-latency=0                                               
        Capabilities: [40] Express Root Port (Slot-), MSI 00                                                       
        Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                            
        Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000                                                
        Capabilities: [a0] Power Management version 2                                                              
        Capabilities: [100] Virtual Channel <?>                                                                    
        Capabilities: [180] Root Complex Link <?>                                                                  
        Kernel driver in use: pcieport-driver                                                                      
        Kernel modules: shpchp                                                                                     

00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                  
        Bus: primary=00, secondary=0a, subordinate=0c, sec-latency=0                                               
        Capabilities: [40] Express Root Port (Slot-), MSI 00                                                       
        Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                            
        Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000                                                
        Capabilities: [a0] Power Management version 2                                                              
        Capabilities: [100] Virtual Channel <?>                                                                    
        Capabilities: [180] Root Complex Link <?>                                                                  
        Kernel driver in use: pcieport-driver                                                                      
        Kernel modules: shpchp                                                                                     

00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0                                                                  
        Bus: primary=00, secondary=0d, subordinate=11, sec-latency=0                                               
        Capabilities: [40] Express Root Port (Slot-), MSI 00                                                       
        Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+                            
        Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000                                                
        Capabilities: [a0] Power Management version 2                                                              
        Capabilities: [100] Virtual Channel <?>                                                                    
        Capabilities: [180] Root Complex Link <?>                                                                  
        Kernel driver in use: pcieport-driver                                                                      
        Kernel modules: shpchp                                                                                     

00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Hewlett-Packard Company Device 31fe                                                            
        Flags: bus master, medium devsel, latency 0, IRQ 21                                                       
        I/O ports at 1000 [size=32]                                                                               
        Capabilities: [50] PCIe advanced features <?>                                                             
        Kernel driver in use: uhci_hcd                                                                            
        Kernel modules: uhci-hcd                                                                                  

00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Hewlett-Packard Company Device 31fe                                                            
        Flags: bus master, medium devsel, latency 0, IRQ 21                                                       
        I/O ports at 1020 [size=32]                                                                               
        Capabilities: [50] PCIe advanced features <?>                                                             
        Kernel driver in use: uhci_hcd                                                                            
        Kernel modules: uhci-hcd                                                                                  

00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Hewlett-Packard Company Device 31fe                                                            
        Flags: bus master, medium devsel, latency 0, IRQ 21                                                       
        I/O ports at 1040 [size=32]                                                                               
        Capabilities: [50] PCIe advanced features <?>                                                             
        Kernel driver in use: uhci_hcd                                                                            
        Kernel modules: uhci-hcd                                                                                  

00:1d.3 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Hewlett-Packard Company Device 31fe                                                            
        Flags: bus master, medium devsel, latency 0, IRQ 21                                                       
        I/O ports at 1060 [size=32]                                                                               
        Capabilities: [50] PCIe advanced features <?>                                                             
        Kernel driver in use: uhci_hcd                                                                            
        Kernel modules: uhci-hcd                                                                                  

00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) (prog-if 20 [EHCI])
        Subsystem: Hewlett-Packard Company Device 31fe                                                             
        Flags: bus master, medium devsel, latency 0, IRQ 21                                                        
        Memory at fdcf0000 (32-bit, non-prefetchable) [size=1K]                                                    
        Capabilities: [50] Power Management version 2                                                              
        Capabilities: [58] Debug port: BAR=1 offset=00a0                                                           
        Capabilities: [98] PCIe advanced features <?>                                                              
        Kernel driver in use: ehci_hcd                                                                             
        Kernel modules: ehci-hcd                                                                                   

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0                                                
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64                            
        I/O behind bridge: 00002000-00003fff                                                     
        Memory behind bridge: fdd00000-fdefffff                                                  
        Prefetchable memory behind bridge: 00000000d8000000-00000000dfffffff                     
        Capabilities: [50] Subsystem: Hewlett-Packard Company Device 31fe                        

00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
        Flags: bus master, medium devsel, latency 0                                    
        Capabilities: [e0] Vendor Specific Information <?>                             

00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port SATA IDE Controller (rev 02) (prog-if 8f [Master SecP SecO PriP PriO])                                                                                                                                    
        Subsystem: Hewlett-Packard Company Device 31fe                                                                                     
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 20                                                                         
        I/O ports at 1080 [size=8]                                                                                                         
        I/O ports at 1088 [size=4]                                                                                                         
        I/O ports at 1090 [size=8]                                                                                                         
        I/O ports at 1098 [size=4]                                                                                                         
        I/O ports at 10a0 [size=16]                                                                                                        
        I/O ports at 10b0 [size=16]                                                                                                        
        Capabilities: [70] Power Management version 3                                                                                      
        Capabilities: [b0] PCIe advanced features <?>                                                                                      
        Kernel driver in use: ata_piix                                                                                                     
        Kernel modules: ide-pci-generic, ata_generic, ata_piix                                                                             

00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE Controller (rev 02) (prog-if 85 [Master SecO PriO])
        Subsystem: Hewlett-Packard Company Device 31fe                                                                           
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 20                                                               
        I/O ports at 10c0 [size=8]                                                                                               
        I/O ports at 10c8 [size=4]                                                                                               
        I/O ports at 10d0 [size=8]                                                                                               
        I/O ports at 10d8 [size=4]                                                                                               
        I/O ports at 10e0 [size=16]                                                                                              
        I/O ports at 10f0 [size=16]                                                                                              
        Capabilities: [70] Power Management version 3                                                                            
        Capabilities: [b0] PCIe advanced features <?>                                                                            
        Kernel driver in use: ata_piix                                                                                           
        Kernel modules: ide-pci-generic, ata_generic, ata_piix                                                                   

01:02.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) (prog-if 00 [VGA controller])
        Subsystem: Hewlett-Packard Company Device 31fb                                               
        Flags: bus master, stepping, medium devsel, latency 64, IRQ 5                                
        Memory at d8000000 (32-bit, prefetchable) [size=128M]                                        
        I/O ports at 3000 [size=256]                                                                 
        Memory at fdef0000 (32-bit, non-prefetchable) [size=64K]                                     
        [virtual] Expansion ROM at fdd00000 [disabled] [size=128K]                                   
        Capabilities: [50] Power Management version 2                                                

01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
        Subsystem: Hewlett-Packard Company Device 3305                                          
        Flags: medium devsel, IRQ 22                                                            
        I/O ports at 2800 [size=256]                                                            
        Memory at fdee0000 (32-bit, non-prefetchable) [size=512]                                
        Capabilities: [f0] Power Management version 3                                           
        Kernel driver in use: hpwdt                                                             
        Kernel modules: hpwdt                                                                   

01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
        Subsystem: Hewlett-Packard Company Device 3305                                          
        Flags: bus master, medium devsel, latency 64, IRQ 23                                    
        BIST result: 00                                                                         
        I/O ports at 3400 [size=256]                                                            
        Memory at fded0000 (32-bit, non-prefetchable) [size=2K]                                 
        Memory at fdec0000 (32-bit, non-prefetchable) [size=16K]                                
        Memory at fde00000 (32-bit, non-prefetchable) [size=512K]                               
        [virtual] Expansion ROM at fdd20000 [disabled] [size=64K]                               
        Capabilities: [f0] Power Management version 3                                           
        Kernel driver in use: hpilo                                                             
        Kernel modules: hpilo                                                                   

01:04.4 USB Controller: Hewlett-Packard Company Proliant iLO2 virtual USB controller (prog-if 00 [UHCI])
        Subsystem: Hewlett-Packard Company Device 3305                                                  
        Flags: bus master, medium devsel, latency 64, IRQ 23                                            
        I/O ports at 3800 [size=32]                                                                     
        Capabilities: [f0] Power Management version 3
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci-hcd

01:04.6 IPMI SMIC interface: Hewlett-Packard Company Proliant iLO2 virtual UART (prog-if 01)
        Subsystem: Hewlett-Packard Company Device 3305
        Flags: medium devsel, IRQ 22
        Memory at fddf0000 (32-bit, non-prefetchable) [size=256]
        Capabilities: [f0] Power Management version 3
        Kernel driver in use: ipmi_si
        Kernel modules: ipmi_si

02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev b5) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
        Memory behind bridge: fdf00000-fdffffff
        Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00
        Capabilities: [90] PCI-X bridge device
        Capabilities: [b0] Power Management version 2
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [14c] Power Budgeting <?>
        Kernel modules: shpchp

03:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)
        Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server Adapter
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 1272
        Memory at fdff0000 (64-bit, non-prefetchable) [size=64K]
        Memory at fdfe0000 (64-bit, non-prefetchable) [size=64K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] PCI-X non-bridge device
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data <?>
        Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable+
        Kernel driver in use: tg3
        Kernel modules: tg3

03:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)
        Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server Adapter
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
        Memory at fdfd0000 (64-bit, non-prefetchable) [size=64K]
        Memory at fdfc0000 (64-bit, non-prefetchable) [size=64K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] PCI-X non-bridge device
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data <?>
        Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable-
        Kernel driver in use: tg3
        Kernel modules: tg3

Comment 2 Anonymous Emailer 2009-03-15 14:38:30 UTC

Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sun, 15 Mar 2009 07:23:00 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12877
> 
>            Summary: tg3: eth0 transit timed out, resetting -> dead NIC
>            Product: Drivers
>            Version: 2.5
>      KernelVersion: 2.6.28.7
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: high
>           Priority: P1
>          Component: Network
>         AssignedTo: drivers_network@kernel-bugs.osdl.org
>         ReportedBy: berni@birkenwald.de
> 
> 
> Latest working kernel version: none
> Earliest failing kernel version: 2.6.28.1
> Distribution: Debian Lenny
> Hardware Environment: HP DL320G5p
> Software Environment: Debian Lenny host for KVM VMs
> Problem Description:
> 
> Every couple of weeks the network of my colo box dies with the following
> message:
> 
> [784060.816020] ------------[ cut here ]------------
> [784060.869153] WARNING: at net/sched/sch_generic.c:226
> dev_watchdog+0x121/0x1b8()
> [784060.953146] NETDEV WATCHDOG: eth0 (tg3): transmit timed out
> [784061.018138] Modules linked in: esp6 xfrm6_mode_tunnel authenc esp4
> xfrm4_mode_tunnel tun kvm_intel kvm xt_NOTRACK ip6table_raw ip6t_LOG
> nf_conntrack_ipv6 ip6table_filter ip6_tables xt_physdev ipt_LOG xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_hashlimit
> iptable_filter ip_tables x_tables bridge stp llc deflate zlib_deflate
> zlib_inflate ctr twofish twofish_common camellia serpent blowfish des_generic
> cbc aes_x86_64 aes_generic xcbc sha256_generic sha1_generic crypto_null
> af_key
> dm_crypt ipv6 coretemp loop ipmi_si ipmi_msghandler hpilo hpwdt pcspkr shpchp
> pci_hotplug container button psmouse serio_raw evdev ext3 jbd dm_mirror
> dm_region_hash dm_log dm_snapshot dm_mod raid1 md_mod sg sd_mod sr_mod cdrom
> usbhid hid ata_piix ata_generic libata scsi_mod ide_pci_generic ide_core
> ehci_hcd tg3 libphy uhci_hcd thermal processor fan thermal_sys
> [784061.891133] Pid: 0, comm: swapper Not tainted 2.6.28.7 #1
> [784061.954129] Call Trace:
> [784061.983133]  <IRQ>  [<ffffffff802398aa>] warn_slowpath+0xb4/0xda
> [784062.053147]  [<ffffffffa0269998>] dst_output+0x0/0xb [ipv6]
> [784062.118130]  [<ffffffff803e35a8>] nf_hook_slow+0x62/0xc3
> [784062.180139]  [<ffffffffa0269998>] dst_output+0x0/0xb [ipv6]
> [784062.245126]  [<ffffffff80332437>] __next_cpu+0x19/0x26
> [784062.305124]  [<ffffffff80212a61>] read_tsc+0xa/0x1f
> [784062.362126]  [<ffffffff80251c69>] getnstimeofday+0x52/0xac
> [784062.426126]  [<ffffffff803d9bd1>] dev_watchdog+0x121/0x1b8
> [784062.490124]  [<ffffffff8025015f>] sched_clock_tick+0x8a/0x92
> [784062.556124]  [<ffffffff803d9ab0>] dev_watchdog+0x0/0x1b8
> [784062.618123]  [<ffffffff80241f50>] run_timer_softirq+0x198/0x21a
> [784062.687118]  [<ffffffff80251c69>] getnstimeofday+0x52/0xac
> [784062.751117]  [<ffffffff8023e61a>] __do_softirq+0x83/0x143
> [784062.814116]  [<ffffffff8020d6ec>] call_softirq+0x1c/0x28
> [784062.876119]  [<ffffffff8020ecd0>] do_softirq+0x3c/0x81
> [784062.936114]  [<ffffffff8023e338>] irq_exit+0x3f/0x83
> [784062.994139]  [<ffffffff8021be99>] smp_apic_timer_interrupt+0x92/0xab
> [784063.068116]  [<ffffffff8020cef8>] apic_timer_interrupt+0x88/0x90
> [784063.138109]  <EOI>  [<ffffffffa03c4a20>] handle_halt+0x0/0x12 [kvm_intel]
> [784063.217116]  [<ffffffff802134fe>] mwait_idle+0x3c/0x46
> [784063.277113]  [<ffffffff8020b0bd>] cpu_idle+0x51/0x92
> [784063.335127] ---[ end trace 444b547394c96982 ]---
> [784063.389142] tg3: eth0: transmit timed out, resetting
> [784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff] MAC_RX_STATUS[ffffffff]
> [784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]
> [784063.706035] tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
> [784063.875340] tg3: tg3_stop_block timed out, ofs=2000 enable_bit=2
> [784064.044372] tg3: tg3_stop_block timed out, ofs=2400 enable_bit=2
> [784064.213191] tg3: tg3_stop_block timed out, ofs=2800 enable_bit=2
> [784064.382454] tg3: tg3_stop_block timed out, ofs=3000 enable_bit=2
> [784064.551295] tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> [784064.720269] tg3: tg3_stop_block timed out, ofs=1800 enable_bit=2
> [784064.889183] tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> [784065.057321] tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
> [784065.226318] tg3: tg3_stop_block timed out, ofs=1000 enable_bit=2
> [784065.395423] tg3: tg3_stop_block timed out, ofs=1c00 enable_bit=2
> [784065.564199] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not
> clear MAC_TX_MODE=ffffffff
> [784065.769278] tg3: tg3_stop_block timed out, ofs=3c00 enable_bit=2
> [784065.938319] tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> [784067.283239] tg3: eth0: No firmware running.
> [784068.533652] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not
> clear MAC_TX_MODE=ffffffff
> [784081.605984] tg3: eth0: Link is down.
> 
> When it happens I either have to reboot the system or rmmod/modprobe tg3 to
> get
> it working again. The interface affected is the routed upstream port of the
> system, the system doesn't do much more than to route/firewall to an internal
> bridge where several KVM VMs are connected to. eth0 has a shared physical
> port
> with the on-board iLO2, which is still reachable when the problem happens.
> The
> switchport bounces a couple of times though.
> 
> Steps to reproduce:
>

Comment 3 Michael Chan 2009-03-16 14:29:55 UTC

On Sun, 2009-03-15 at 14:32 -0700, Andrew Morton wrote:
> > [784063.389142] tg3: eth0: transmit timed out, resetting
> > [784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff] MAC_RX_STATUS[ffffffff]
> > [784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]

At the time of tx timeout, the registers all return 0xffffffff.  Does
the subsequent reset bring the device back?  If the device is brought
back, there should be a link up message and traffic should resume.  If
not, please provide lspci -vvvxxx on the eth0 device after the failure.

Also, when one ethernet port fails, does the other port (from the same
dual port device) function ok?

Thanks.

Comment 4 Bernhard Schmidt 2009-03-16 15:47:51 UTC

On 16.03.2009 22:23, Michael Chan wrote:
> On Sun, 2009-03-15 at 14:32 -0700, Andrew Morton wrote:
>>> [784063.389142] tg3: eth0: transmit timed out, resetting
>>> [784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff] MAC_RX_STATUS[ffffffff]
>>> [784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]
>
> At the time of tx timeout, the registers all return 0xffffffff.  Does
> the subsequent reset bring the device back?  If the device is brought
> back, there should be a link up message and traffic should resume.  If
> not, please provide lspci -vvvxxx on the eth0 device after the failure.

The port does not pass traffic until I rmmod/modprobe the driver or 
reset the whole system, it doesn't recover by itself. "[784081.605984] 
tg3: eth0: Link is down." is the last message I see.

I'll send you the lspci output as soon as it happens again.

> Also, when one ethernet port fails, does the other port (from the same
> dual port device) function ok?

I'm not exactly sure about that, as nowadays there is nothing connected 
to it anymore. However, when the problem first occured there was, and 
I'm pretty sure the second port was okay.

Bernhard

Comment 5 Bernhard Schmidt 2009-03-17 15:10:21 UTC

On 16.03.2009 22:23, Michael Chan wrote:
> On Sun, 2009-03-15 at 14:32 -0700, Andrew Morton wrote:
>>> [784063.389142] tg3: eth0: transmit timed out, resetting
>>> [784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff] MAC_RX_STATUS[ffffffff]
>>> [784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]
>
> At the time of tx timeout, the registers all return 0xffffffff.  Does
> the subsequent reset bring the device back?  If the device is brought
> back, there should be a link up message and traffic should resume.  If
> not, please provide lspci -vvvxxx on the eth0 device after the failure.

Attached, both after the crash (tg3.crashed) and after I reloaded the 
module (tg3.reloaded). Additional info, ifdown/ifup does not fix the 
situation, both take pretty long

# ifdown eth0
tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear 
MAC_TX_MODE=ffffffff
# ifup eth0
tg3 0000:03:04.0: irq 1272 for MSI/MSI-X
ADDRCONF(NETDEV_UP): eth0: link is not ready

and it stays dead.

# rmmod tg3
tg3 0000:03:04.1: PCI INT B disabled
tg3 0000:03:04.0: PCI INT A disabled
# modprobe tg3
tg3.c:v3.94 (August 14, 2008)
tg3 0000:03:04.0: enabling device (0000 -> 0002)
tg3 0000:03:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
eth0: Tigon3 [partno(N/A) rev 9003 PHY(5714)] (PCIX:133MHz:64-bit) 
10/100/1000Base-T Ethernet 00:21:5a:99:0a:28
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] WireSpeed[1] TSOcap[1]
eth0: dma_rwctrl[76148000] dma_mask[40-bit]
tg3 0000:03:04.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
eth1: Tigon3 [partno(N/A) rev 9003 PHY(5714)] (PCIX:133MHz:64-bit) 
10/100/1000Base-T Ethernet 00:21:5a:99:0a:29
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[1] TSOcap[1]
eth1: dma_rwctrl[76148000] dma_mask[40-bit]
# ifup eth0
ADDRCONF(NETDEV_UP): eth0: link is not ready
device eth0 entered promiscuous mode
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

and I'm up and running again.

> Also, when one ethernet port fails, does the other port (from the same
> dual port device) function ok?

Still no clue about that, I need to find someone who can connect 
something there.

Bernhard

Comment 6 Michael Chan 2009-03-17 16:37:34 UTC

On Tue, 2009-03-17 at 15:09 -0700, Bernhard Schmidt wrote:
> Attached, both after the crash (tg3.crashed) and after I reloaded the 
> module (tg3.reloaded). Additional info, ifdown/ifup does not fix the 
> situation, both take pretty long
> 

Thanks for the information.  The memory enable bit in the PCI command
register was cleared during tx_timeout.  That's why all the registers
were reading 0xffffffff.  The tx_timeout code in tg3 would not be able
to reset the chip if that bit was cleared.  We need to find out why that
bit was cleared.  We should also enhance the tx timeout code so that it
can recover more completely even if the memory enable bit is cleared.

Thanks.

Comment 7 Matt Carlson 2009-03-19 10:00:10 UTC

Hi Bernhard.  I talked to Michael about this and we'd like you to try
two things.

1) Can you disable iLo2 and see if you can still reproduce the problem?
2) Can you apply the following patch to get more information on when
   MMIO gets disabled?


diff -Nrup 1/drivers/net/tg3.c 2/drivers/net/tg3.c
--- 1/drivers/net/tg3.c	2009-02-20 14:41:27.000000000 -0800
+++ 2/drivers/net/tg3.c	2009-03-19 09:51:17.000000000 -0700
@@ -7799,6 +7799,12 @@ static void tg3_timer(unsigned long __op
 
 	/* This part only runs once per second. */
 	if (!--tp->timer_counter) {
+		u16 pci_cmd;
+
+		pci_read_config_word(tp->pdev, PCI_COMMAND, &pci_cmd);
+		if (!(pci_cmd & PCI_COMMAND_MEMORY))
+			printk( KERN_WARNING "PCI Memory Mapped IO Disabled!!!!\n" );
+
 		if (tp->tg3_flags2 & TG3_FLG2_5705_PLUS)
 			tg3_periodic_fetch_stats(tp);
 

On Tue, Mar 17, 2009 at 03:09:03PM -0700, Bernhard Schmidt wrote:
> On 16.03.2009 22:23, Michael Chan wrote:
> > On Sun, 2009-03-15 at 14:32 -0700, Andrew Morton wrote:
> >>> [784063.389142] tg3: eth0: transmit timed out, resetting
> >>> [784063.447106] tg3: DEBUG: MAC_TX_STATUS[ffffffff]
> MAC_RX_STATUS[ffffffff]
> >>> [784063.524104] tg3: DEBUG: RDMAC_STATUS[ffffffff] WDMAC_STATUS[ffffffff]
> >
> > At the time of tx timeout, the registers all return 0xffffffff.  Does
> > the subsequent reset bring the device back?  If the device is brought
> > back, there should be a link up message and traffic should resume.  If
> > not, please provide lspci -vvvxxx on the eth0 device after the failure.
> 
> Attached, both after the crash (tg3.crashed) and after I reloaded the 
> module (tg3.reloaded). Additional info, ifdown/ifup does not fix the 
> situation, both take pretty long
> 
> # ifdown eth0
> tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear 
> MAC_TX_MODE=ffffffff
> # ifup eth0
> tg3 0000:03:04.0: irq 1272 for MSI/MSI-X
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> 
> and it stays dead.
> 
> # rmmod tg3
> tg3 0000:03:04.1: PCI INT B disabled
> tg3 0000:03:04.0: PCI INT A disabled
> # modprobe tg3
> tg3.c:v3.94 (August 14, 2008)
> tg3 0000:03:04.0: enabling device (0000 -> 0002)
> tg3 0000:03:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> eth0: Tigon3 [partno(N/A) rev 9003 PHY(5714)] (PCIX:133MHz:64-bit) 
> 10/100/1000Base-T Ethernet 00:21:5a:99:0a:28
> eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] WireSpeed[1] TSOcap[1]
> eth0: dma_rwctrl[76148000] dma_mask[40-bit]
> tg3 0000:03:04.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
> eth1: Tigon3 [partno(N/A) rev 9003 PHY(5714)] (PCIX:133MHz:64-bit) 
> 10/100/1000Base-T Ethernet 00:21:5a:99:0a:29
> eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[1] TSOcap[1]
> eth1: dma_rwctrl[76148000] dma_mask[40-bit]
> # ifup eth0
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> device eth0 entered promiscuous mode
> tg3: eth0: Link is up at 100 Mbps, full duplex.
> tg3: eth0: Flow control is off for TX and off for RX.
> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> 
> and I'm up and running again.
> 
> > Also, when one ethernet port fails, does the other port (from the same
> > dual port device) function ok?
> 
> Still no clue about that, I need to find someone who can connect 
> something there.
> 
> Bernhard

Content-Description: tg3.crashed
> 00:00.0 Host bridge: Intel Corporation 3200/3210 Chipset DRAM Controller (rev
> 01)
>       Subsystem: Hewlett-Packard Company Device 31f9
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ >SERR- <PERR- INTx-
>       Latency: 0
>       Capabilities: [e0] Vendor Specific Information <?>
> 00: 86 80 f0 29 46 01 90 20 01 00 00 06 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 f9 31
> 30: 00 00 00 00 e0 00 00 00 00 00 00 00 00 00 00 00
> 40: 01 90 d1 fe 00 00 00 00 01 40 d1 fe 00 00 00 00
> 50: 00 00 02 00 03 20 00 00 00 00 00 00 00 00 00 00
> 60: 01 00 00 e0 00 00 00 00 01 80 d1 fe 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 30 11 31 13 11 31 33 00 ff 03 00 00 00 0a 38 00
> a0: 30 00 00 0c 00 00 00 c0 00 00 00 c0 00 00 00 c0
> b0: 00 c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 03 00 00 00
> d0: 00 00 00 00 00 00 00 00 49 00 00 00 00 00 00 8c
> e0: 09 00 0c 91 64 c8 fc c0 43 44 0b 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 90 0f 01 00 00 00 00 00
> 
> 00:01.0 PCI bridge: Intel Corporation 3200/3210 Chipset Host-Primary PCI
> Express Bridge (rev 01) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=15, subordinate=1a, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9
>       Capabilities: [80] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4169
>       Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>               LnkCap: Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency
> L0 <512ns, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot+
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl+
>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
> LinkChg-
>                       Control: AttnInd Unknown, PwrInd Unknown, Power-
> Interlock-
>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                       Changed: MRL- PresDet- LinkState-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [140] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 f1 29 47 05 10 00 01 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 15 1a 00 f0 00 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 88 00 00 00 00 00 00 00 ff 01 03 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02
> 80: 01 90 03 c8 08 00 00 00 0d 80 00 00 3c 10 f9 31
> 90: 05 a0 01 00 0c 30 e0 fe 69 41 00 00 00 00 00 00
> a0: 10 00 42 01 00 80 00 00 06 00 00 00 01 3d 21 02
> b0: 40 00 01 10 00 00 04 00 00 00 00 00 0e 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 01 00 03 e1 3f 80 90 0f 01 00 00 f0 00 f0
> 
> 00:06.0 PCI bridge: Intel Corporation 3210 Chipset Host-Secondary PCI Express
> Bridge (rev 01) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=12, subordinate=14, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9
>       Capabilities: [80] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4171
>       Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>               LnkCap: Port #3, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency
> L0 <512ns, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot+
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl+
>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
> LinkChg-
>                       Control: AttnInd Unknown, PwrInd Unknown, Power-
> Interlock-
>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                       Changed: MRL- PresDet- LinkState-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [140] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 f9 29 47 05 10 00 01 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 12 14 00 f0 00 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 88 00 00 00 00 00 00 00 ff 01 03 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02
> 80: 01 90 03 c8 08 00 00 00 0d 80 00 00 3c 10 f9 31
> 90: 05 a0 01 00 0c 30 e0 fe 71 41 00 00 00 00 00 00
> a0: 10 00 42 01 00 80 00 00 06 00 00 00 81 3c 21 03
> b0: 40 00 01 10 00 00 04 00 00 00 00 00 0e 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 01 00 03 e1 3f 80 90 0f 01 00 00 00 00 00
> 
> 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fdf00000-fdffffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
> DLActive+ BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4179
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 40 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 02 03 00 f0 00 00 20
> 20: f0 fd f0 fd f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 21 4c 11 01
> 50: 00 00 21 30 60 00 00 00 00 00 48 01 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 79 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=04, subordinate=06, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4181
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 44 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 04 06 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 03 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 03
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 81 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4189
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 46 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 07 09 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 04 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 04
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 89 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=0a, subordinate=0c, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4191
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 48 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 0a 0c 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 05
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 91 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=0d, subordinate=11, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #6, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4199
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 4a 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 0d 11 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 06
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 99 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #1 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin A routed to IRQ 21
>       Region 4: I/O ports at 1000 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 34 29 05 00 90 02 02 00 03 0c 00 00 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 01 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #2 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 21
>       Region 4: I/O ports at 1020 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 35 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 21 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 02 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #3 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin C routed to IRQ 21
>       Region 4: I/O ports at 1040 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 36 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 41 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 03 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.3 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #6 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin D routed to IRQ 21
>       Region 4: I/O ports at 1060 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 39 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 61 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 04 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
> Controller #1 (rev 02) (prog-if 20 [EHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin A routed to IRQ 21
>       Region 0: Memory at fdcf0000 (32-bit, non-prefetchable) [size=1K]
>       Capabilities: [50] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [58] Debug port: BAR=1 offset=00a0
>       Capabilities: [98] PCIe advanced features <?>
>       Kernel driver in use: ehci_hcd
>       Kernel modules: ehci-hcd
> 00: 86 80 3a 29 46 01 90 02 02 20 03 0c 00 00 00 00
> 10: 00 00 cf fd 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 01 58 c2 c9 00 00 00 00 0a 98 a0 20 00 00 00 00
> 60: 20 20 ff 01 00 00 00 00 01 00 00 01 00 00 08 c0
> 70: 00 00 cf 3f 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 13 00 06 03 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 aa ff 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 88 85 40 00 86 0f 02 00 06 17 02 20
> 
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01
> [Subtractive decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>       I/O behind bridge: 00002000-00003fff
>       Memory behind bridge: fdd00000-fdefffff
>       Prefetchable memory behind bridge: 00000000d8000000-00000000dfffffff
>       Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [50] Subsystem: Hewlett-Packard Company Device 31fe
> 00: 86 80 4e 24 47 01 10 00 92 01 04 06 00 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 01 01 40 20 30 80 22
> 20: d0 fd e0 fd 01 d8 f1 df 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 ff 00 0b 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 12 00 00
> 50: 0d 00 00 00 3c 10 fe 31 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface
> Controller (rev 02)
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Capabilities: [e0] Vendor Specific Information <?>
> 00: 86 80 16 29 47 01 10 02 02 00 01 06 00 00 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 e0 00 00 00 00 00 00 00 00 00 00 00
> 40: 01 09 00 00 80 00 00 00 01 08 00 00 10 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 8b 8a 80 80 d0 00 00 00 87 85 8a 8b f8 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 09 34 01 03 0c 00 01 06 7c 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 02 00 00 08 02 00 00 13 00 00 00 00 03 00 00
> b0: 00 00 f0 00 00 00 00 00 00 40 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 67 45 00 00 cf ff 00 00 01 00 00 00
> e0: 09 00 0c 10 00 00 c4 02 60 00 00 00 00 00 00 00
> f0: 01 c0 d1 fe 44 f2 0e 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port
> SATA IDE Controller (rev 02) (prog-if 8f [Master SecP SecO PriP PriO])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 20
>       Region 0: I/O ports at 1080 [size=8]
>       Region 1: I/O ports at 1088 [size=4]
>       Region 2: I/O ports at 1090 [size=8]
>       Region 3: I/O ports at 1098 [size=4]
>       Region 4: I/O ports at 10a0 [size=16]
>       Region 5: I/O ports at 10b0 [size=16]
>       Capabilities: [70] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [b0] PCIe advanced features <?>
>       Kernel driver in use: ata_piix
>       Kernel modules: ide-pci-generic, ata_generic, ata_piix
> 00: 86 80 20 29 47 00 b0 02 02 8f 01 01 00 00 00 00
> 10: 81 10 00 00 89 10 00 00 91 10 00 00 99 10 00 00
> 20: a1 10 00 00 b1 10 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 70 00 00 00 00 00 00 00 07 02 00 00
> 40: 00 80 00 80 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 01 b0 03 00 08 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 0f 03 80 01 80 00 00 00 00 00 00 00 00 00
> a0: ac 00 00 00 0a 00 12 00 00 00 00 00 00 00 00 00
> b0: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE
> Controller (rev 02) (prog-if 85 [Master SecO PriO])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 20
>       Region 0: I/O ports at 10c0 [size=8]
>       Region 1: I/O ports at 10c8 [size=4]
>       Region 2: I/O ports at 10d0 [size=8]
>       Region 3: I/O ports at 10d8 [size=4]
>       Region 4: I/O ports at 10e0 [size=16]
>       Region 5: I/O ports at 10f0 [size=16]
>       Capabilities: [70] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [b0] PCIe advanced features <?>
>       Kernel driver in use: ata_piix
>       Kernel modules: ide-pci-generic, ata_generic, ata_piix
> 00: 86 80 26 29 47 00 b0 02 02 85 01 01 00 00 00 00
> 10: c1 10 00 00 c9 10 00 00 d1 10 00 00 d9 10 00 00
> 20: e1 10 00 00 f1 10 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 70 00 00 00 00 00 00 00 07 02 00 00
> 40: 00 80 00 80 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 01 b0 03 00 08 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 03 01 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 01:02.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
> (prog-if 00 [VGA controller])
>       Subsystem: Hewlett-Packard Company Device 31fb
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping+ SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64 (2000ns min), Cache Line Size: 64 bytes
>       Interrupt: pin A routed to IRQ 5
>       Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M]
>       Region 1: I/O ports at 3000 [size=256]
>       Region 2: Memory at fdef0000 (32-bit, non-prefetchable) [size=64K]
>       [virtual] Expansion ROM at fdd00000 [disabled] [size=128K]
>       Capabilities: [50] Power Management version 2
>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> 00: 02 10 5e 51 87 01 90 02 02 00 00 03 10 40 00 00
> 10: 08 00 00 d8 01 30 00 00 00 00 ef fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fb 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 08 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fb 31
> 50: 01 00 02 06 00 00 00 00 02 50 20 00 00 02 00 2f
> 60: 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out
> Controller (rev 03)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 22
>       Region 0: I/O ports at 2800 [size=256]
>       Region 1: Memory at fdee0000 (32-bit, non-prefetchable) [size=512]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: hpwdt
>       Kernel modules: hpwdt
> 00: 11 0e 03 b2 03 01 90 02 03 00 80 08 00 00 80 00
> 10: 01 28 00 00 00 00 ee fd 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0a 01 00 00
> 40: 67 fd 00 00 11 00 00 00 00 00 00 00 06 02 06 02
> 50: 00 00 00 00 00 00 00 00 ef 39 fd 77 01 00 00 5a
> 60: 00 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 1a 00 88 03 43 50 51 00
> 90: 30 00 00 00 02 00 00 5f 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 02 00 00 00 00 00 04 04 c1 02
> b0: 00 0a 00 00 00 00 00 00 b7 01 10 51 84 00 03 09
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 01 00 00 00 00 13 21 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out 
> Processor (rev 03)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64, Cache Line Size: 64 bytes
>       Interrupt: pin B routed to IRQ 23
>       BIST result: 00
>       Region 0: I/O ports at 3400 [size=256]
>       Region 1: Memory at fded0000 (32-bit, non-prefetchable) [size=2K]
>       Region 2: Memory at fdec0000 (32-bit, non-prefetchable) [size=16K]
>       Region 3: Memory at fde00000 (32-bit, non-prefetchable) [size=512K]
>       [virtual] Expansion ROM at fdd20000 [disabled] [size=64K]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: hpilo
>       Kernel modules: hpilo
> 00: 11 0e 04 b2 17 01 90 02 03 00 80 08 10 40 80 80
> 10: 01 34 00 00 00 00 ed fd 00 00 ec fd 00 00 e0 fd
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0b 02 00 00
> 40: 04 0d 00 00 00 00 00 00 00 00 00 00 0a 01 15 12
> 50: 03 e0 ff 00 05 00 40 00 04 00 00 00 00 00 e0 00
> 60: 00 00 00 00 00 00 00 00 8b 60 56 05 59 28 82 f8
> 70: 00 08 00 01 03 03 c0 00 00 00 00 00 00 00 00 00
> 80: fe 01 00 00 62 6a 06 00 6b 63 9b 14 88 48 00 00
> 90: 00 00 00 00 03 03 00 00 00 00 00 00 00 00 00 00
> a0: 14 03 02 00 80 00 00 00 20 00 00 00 00 00 00 00
> b0: 10 00 00 00 00 00 e7 81 ff fd 0e 0d 00 00 00 00
> c0: a3 0c 00 00 00 00 07 00 a5 0c 00 00 00 c0 07 00
> d0: 00 00 00 00 00 00 00 00 9a 17 00 00 00 00 00 00
> e0: 00 00 00 00 0b 00 00 00 00 00 00 00 0b 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.4 USB Controller: Hewlett-Packard Company Proliant iLO2 virtual USB
> controller (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64
>       Interrupt: pin B routed to IRQ 23
>       Region 4: I/O ports at 3800 [size=32]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 3c 10 00 33 45 01 90 02 00 00 03 0c 00 40 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 01 38 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0b 02 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Proliant iLO2 virtual
> UART (prog-if 01)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 22
>       Region 0: Memory at fddf0000 (32-bit, non-prefetchable) [size=256]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: ipmi_si
>       Kernel modules: ipmi_si
> 00: 3c 10 02 33 02 00 90 02 00 01 07 0c 00 00 80 00
> 10: 00 00 df fd 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0a 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev b5)
> (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fdf00000-fdffffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [60] Express (v1) PCI/PCI-X Bridge, MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1
> <16us
>                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> BrConfRtry-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency
> L0 <4us, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot-
>               LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>       Capabilities: [90] PCI-X bridge device
>               Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD-
> Freq=133MHz
>               Status: Dev=02:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
>               Upstream: Capacity=0 CommitmentLimit=0
>               Downstream: Capacity=0 CommitmentLimit=0
>       Capabilities: [b0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Advanced Error Reporting <?>
>       Capabilities: [14c] Power Budgeting <?>
>       Kernel modules: shpchp
> 00: 66 11 03 01 47 01 10 00 b5 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 02 03 03 40 f1 01 00 22
> 20: f0 fd f0 fd f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 60 00 00 00 00 00 00 00 60 00 03 00
> 40: 00 00 00 00 00 00 00 01 00 00 0c 20 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 c0 00 00 c0
> 60: 10 90 71 00 a0 09 00 00 06 00 10 00 41 6c 01 00
> 70: 00 00 21 10 44 08 00 00 00 00 00 00 00 00 00 00
> 80: 01 00 00 00 20 20 30 2b 92 08 18 20 02 60 00 fb
> 90: 07 b0 c3 00 00 02 00 00 00 00 00 00 00 00 00 00
> a0: 90 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 01 00 c2 c9 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 03:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit
> Ethernet (rev a3)
>       Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server
> Adapter
>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 1272
>       Region 0: Memory at fdff0000 (64-bit, non-prefetchable) [disabled]
> [size=64K]
>       Region 2: Memory at fdfe0000 (64-bit, non-prefetchable) [disabled]
> [size=64K]
>       Expansion ROM at <ignored> [disabled]
>       Capabilities: [40] PCI-X non-bridge device
>               Command: DPERE- ERO- RBC=512 OST=1
>               Status: Dev=03:04.0 64bit+ 133MHz+ SCD- USC- DC=simple
> DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
>       Capabilities: [48] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
>       Capabilities: [50] Vital Product Data <?>
>       Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
> Enable+
>               Address: 00000000fee0300c  Data: 41d1
>       Kernel driver in use: tg3
>       Kernel modules: tg3
> 00: e4 14 78 16 00 00 b0 02 a3 00 00 02 10 40 80 00
> 10: 04 00 ff fd 00 00 00 00 04 00 fe fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 3e 70
> 30: 00 00 e3 82 40 00 00 00 00 00 00 00 0b 01 40 00
> 40: 07 48 00 00 20 03 43 04 01 50 02 c0 00 21 00 64
> 50: 03 58 04 00 3f 61 00 10 05 00 87 00 0c 30 e0 fe
> 60: 00 00 00 00 d1 41 00 00 9a 02 03 90 00 00 00 00
> 70: 6a 10 00 00 c7 00 00 00 50 00 00 00 00 00 00 00
> 80: 03 58 04 00 40 00 00 40 00 00 01 00 fe d0 08 00
> 90: 01 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 03:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit
> Ethernet (rev a3)
>       Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server
> Adapter
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64 (16000ns min), Cache Line Size: 64 bytes
>       Interrupt: pin B routed to IRQ 17
>       Region 0: Memory at fdfd0000 (64-bit, non-prefetchable) [size=64K]
>       Region 2: Memory at fdfc0000 (64-bit, non-prefetchable) [size=64K]
>       Expansion ROM at <ignored> [disabled]
>       Capabilities: [40] PCI-X non-bridge device
>               Command: DPERE- ERO- RBC=512 OST=1
>               Status: Dev=03:04.1 64bit+ 133MHz+ SCD- USC- DC=simple
> DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
>       Capabilities: [48] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=1 PME-
>       Capabilities: [50] Vital Product Data <?>
>       Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
> Enable-
>               Address: 00000000fee0300c  Data: 41c9
>       Kernel driver in use: tg3
>       Kernel modules: tg3
> 00: e4 14 78 16 46 01 b0 02 a3 00 00 02 10 40 80 00
> 10: 04 00 fd fd 00 00 00 00 04 00 fc fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 3e 70
> 30: 00 00 6f ba 40 00 00 00 00 00 00 00 0a 02 40 00
> 40: 07 48 00 00 21 03 43 04 01 50 02 c0 00 20 00 64
> 50: 03 58 e8 00 6b ee 9b 7d 05 00 86 00 0c 30 e0 fe
> 60: 00 00 00 00 c9 41 00 00 9a 02 03 90 00 80 14 76
> 70: ea 10 00 00 c7 00 00 00 14 04 00 00 00 00 00 00
> 80: 29 0a 99 5a 2a 00 00 00 34 00 00 00 fe d0 08 04
> 90: 01 07 00 09 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 

Content-Description: tg3.reloaded
> 00:00.0 Host bridge: Intel Corporation 3200/3210 Chipset DRAM Controller (rev
> 01)
>       Subsystem: Hewlett-Packard Company Device 31f9
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ >SERR- <PERR- INTx-
>       Latency: 0
>       Capabilities: [e0] Vendor Specific Information <?>
> 00: 86 80 f0 29 46 01 90 20 01 00 00 06 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 f9 31
> 30: 00 00 00 00 e0 00 00 00 00 00 00 00 00 00 00 00
> 40: 01 90 d1 fe 00 00 00 00 01 40 d1 fe 00 00 00 00
> 50: 00 00 02 00 03 20 00 00 00 00 00 00 00 00 00 00
> 60: 01 00 00 e0 00 00 00 00 01 80 d1 fe 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 30 11 31 13 11 31 33 00 ff 03 00 00 00 0a 38 00
> a0: 30 00 00 0c 00 00 00 c0 00 00 00 c0 00 00 00 c0
> b0: 00 c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 03 00 00 00
> d0: 00 00 00 00 00 00 00 00 49 00 00 00 00 00 00 8c
> e0: 09 00 0c 91 64 c8 fc c0 43 44 0b 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 90 0f 01 00 00 00 00 00
> 
> 00:01.0 PCI bridge: Intel Corporation 3200/3210 Chipset Host-Primary PCI
> Express Bridge (rev 01) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=15, subordinate=1a, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9
>       Capabilities: [80] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4169
>       Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>               LnkCap: Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency
> L0 <512ns, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot+
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl+
>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
> LinkChg-
>                       Control: AttnInd Unknown, PwrInd Unknown, Power-
> Interlock-
>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                       Changed: MRL- PresDet- LinkState-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [140] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 f1 29 47 05 10 00 01 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 15 1a 00 f0 00 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 88 00 00 00 00 00 00 00 ff 01 03 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02
> 80: 01 90 03 c8 08 00 00 00 0d 80 00 00 3c 10 f9 31
> 90: 05 a0 01 00 0c 30 e0 fe 69 41 00 00 00 00 00 00
> a0: 10 00 42 01 00 80 00 00 06 00 00 00 01 3d 21 02
> b0: 40 00 01 10 00 00 04 00 00 00 00 00 0e 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 01 00 03 e1 3f 80 90 0f 01 00 00 f0 00 f0
> 
> 00:06.0 PCI bridge: Intel Corporation 3210 Chipset Host-Secondary PCI Express
> Bridge (rev 01) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=12, subordinate=14, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [88] Subsystem: Hewlett-Packard Company Device 31f9
>       Capabilities: [80] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4171
>       Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>               LnkCap: Port #3, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency
> L0 <512ns, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot+
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl+
>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
> LinkChg-
>                       Control: AttnInd Unknown, PwrInd Unknown, Power-
> Interlock-
>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                       Changed: MRL- PresDet- LinkState-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [140] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 f9 29 47 05 10 00 01 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 12 14 00 f0 00 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 88 00 00 00 00 00 00 00 ff 01 03 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02
> 80: 01 90 03 c8 08 00 00 00 0d 80 00 00 3c 10 f9 31
> 90: 05 a0 01 00 0c 30 e0 fe 71 41 00 00 00 00 00 00
> a0: 10 00 42 01 00 80 00 00 06 00 00 00 81 3c 21 03
> b0: 40 00 01 10 00 00 04 00 00 00 00 00 0e 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 01 00 03 e1 3f 80 90 0f 01 00 00 00 00 00
> 
> 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fdf00000-fdffffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
> DLActive+ BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4179
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 40 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 02 03 00 f0 00 00 20
> 20: f0 fd f0 fd f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 21 4c 11 01
> 50: 00 00 21 30 60 00 00 00 00 00 48 01 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 79 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=04, subordinate=06, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4181
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 44 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 04 06 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 03 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 03
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 81 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4189
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 46 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 07 09 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 04 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 04
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 89 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=0a, subordinate=0c, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4191
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 48 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 0a 0c 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 05
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 91 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6
> (rev 02) (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=00, secondary=0d, subordinate=11, sec-latency=0
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fff00000-000fffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [40] Express (v1) Root Port (Slot-), MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                       ExtTag- RBE+ FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #6, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
> L0 <1us, L1 <4us
>                       ClockPM- Suprise- LLActRep+ BwNot-
>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>               RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+
> CRSVisible-
>               RootCap: CRSVisible-
>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>       Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0
> Enable+
>               Address: fee0300c  Data: 4199
>       Capabilities: [90] Subsystem: Gammagraphx, Inc. Device 0000
>       Capabilities: [a0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Virtual Channel <?>
>       Capabilities: [180] Root Complex Link <?>
>       Kernel driver in use: pcieport-driver
>       Kernel modules: shpchp
> 00: 86 80 4a 29 47 05 10 00 02 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 0d 11 00 f0 00 00 20
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 03 00
> 40: 10 80 41 00 00 80 00 00 06 00 10 00 11 4c 11 06
> 50: 00 00 01 10 60 00 00 00 00 00 40 00 0e 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 90 01 00 0c 30 e0 fe 99 41 00 00 00 00 00 00
> 90: 0d a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 01 00 02 c8 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 11 08 00 00 00 00
> e0: 00 00 c7 00 06 07 08 00 30 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #1 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin A routed to IRQ 21
>       Region 4: I/O ports at 1000 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 34 29 05 00 90 02 02 00 03 0c 00 00 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 01 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #2 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 21
>       Region 4: I/O ports at 1020 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 35 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 21 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 02 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #3 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin C routed to IRQ 21
>       Region 4: I/O ports at 1040 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 36 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 41 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 03 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.3 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #6 (rev 02) (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin D routed to IRQ 21
>       Region 4: I/O ports at 1060 [size=32]
>       Capabilities: [50] PCIe advanced features <?>
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 86 80 39 29 05 00 90 02 02 00 03 0c 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 61 10 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 04 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 2f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
> Controller #1 (rev 02) (prog-if 20 [EHCI])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin A routed to IRQ 21
>       Region 0: Memory at fdcf0000 (32-bit, non-prefetchable) [size=1K]
>       Capabilities: [50] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [58] Debug port: BAR=1 offset=00a0
>       Capabilities: [98] PCIe advanced features <?>
>       Kernel driver in use: ehci_hcd
>       Kernel modules: ehci-hcd
> 00: 86 80 3a 29 46 01 90 02 02 20 03 0c 00 00 00 00
> 10: 00 00 cf fd 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 01 58 c2 c9 00 00 00 00 0a 98 a0 20 00 00 00 00
> 60: 20 20 ff 01 00 00 00 00 01 00 00 01 00 00 08 c0
> 70: 00 00 cf 3f 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 13 00 06 03 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 aa ff 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 88 85 40 00 86 0f 02 00 06 17 02 20
> 
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01
> [Subtractive decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>       I/O behind bridge: 00002000-00003fff
>       Memory behind bridge: fdd00000-fdefffff
>       Prefetchable memory behind bridge: 00000000d8000000-00000000dfffffff
>       Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [50] Subsystem: Hewlett-Packard Company Device 31fe
> 00: 86 80 4e 24 47 01 10 00 92 01 04 06 00 00 01 00
> 10: 00 00 00 00 00 00 00 00 00 01 01 40 20 30 80 22
> 20: d0 fd e0 fd 01 d8 f1 df 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 ff 00 0b 00
> 40: 00 00 00 00 00 00 00 00 01 00 00 00 00 12 00 00
> 50: 0d 00 00 00 3c 10 fe 31 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface
> Controller (rev 02)
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Capabilities: [e0] Vendor Specific Information <?>
> 00: 86 80 16 29 47 01 10 02 02 00 01 06 00 00 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 e0 00 00 00 00 00 00 00 00 00 00 00
> 40: 01 09 00 00 80 00 00 00 01 08 00 00 10 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 8b 8a 80 80 d0 00 00 00 87 85 8a 8b f8 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 09 34 01 03 0c 00 01 06 7c 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 02 00 00 08 02 00 00 13 00 00 00 00 03 00 00
> b0: 00 00 f0 00 00 00 00 00 00 40 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 67 45 00 00 cf ff 00 00 01 00 00 00
> e0: 09 00 0c 10 00 00 c4 02 60 00 00 00 00 00 00 00
> f0: 01 c0 d1 fe 44 f2 0e 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port
> SATA IDE Controller (rev 02) (prog-if 8f [Master SecP SecO PriP PriO])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 20
>       Region 0: I/O ports at 1080 [size=8]
>       Region 1: I/O ports at 1088 [size=4]
>       Region 2: I/O ports at 1090 [size=8]
>       Region 3: I/O ports at 1098 [size=4]
>       Region 4: I/O ports at 10a0 [size=16]
>       Region 5: I/O ports at 10b0 [size=16]
>       Capabilities: [70] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [b0] PCIe advanced features <?>
>       Kernel driver in use: ata_piix
>       Kernel modules: ide-pci-generic, ata_generic, ata_piix
> 00: 86 80 20 29 47 00 b0 02 02 8f 01 01 00 00 00 00
> 10: 81 10 00 00 89 10 00 00 91 10 00 00 99 10 00 00
> 20: a1 10 00 00 b1 10 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 70 00 00 00 00 00 00 00 07 02 00 00
> 40: 00 80 00 80 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 01 b0 03 00 08 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 0f 03 80 01 80 00 00 00 00 00 00 00 00 00
> a0: ac 00 00 00 0a 00 12 00 00 00 00 00 00 00 00 00
> b0: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE
> Controller (rev 02) (prog-if 85 [Master SecO PriO])
>       Subsystem: Hewlett-Packard Company Device 31fe
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Interrupt: pin B routed to IRQ 20
>       Region 0: I/O ports at 10c0 [size=8]
>       Region 1: I/O ports at 10c8 [size=4]
>       Region 2: I/O ports at 10d0 [size=8]
>       Region 3: I/O ports at 10d8 [size=4]
>       Region 4: I/O ports at 10e0 [size=16]
>       Region 5: I/O ports at 10f0 [size=16]
>       Capabilities: [70] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [b0] PCIe advanced features <?>
>       Kernel driver in use: ata_piix
>       Kernel modules: ide-pci-generic, ata_generic, ata_piix
> 00: 86 80 26 29 47 00 b0 02 02 85 01 01 00 00 00 00
> 10: c1 10 00 00 c9 10 00 00 d1 10 00 00 d9 10 00 00
> 20: e1 10 00 00 f1 10 00 00 00 00 00 00 3c 10 fe 31
> 30: 00 00 00 00 70 00 00 00 00 00 00 00 07 02 00 00
> 40: 00 80 00 80 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 01 b0 03 00 08 00 00 00 00 00 00 00 00 00 00 00
> 80: 05 70 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 03 01 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 13 00 06 03 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 86 0f 02 00 00 00 00 00
> 
> 01:02.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
> (prog-if 00 [VGA controller])
>       Subsystem: Hewlett-Packard Company Device 31fb
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping+ SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64 (2000ns min), Cache Line Size: 64 bytes
>       Interrupt: pin A routed to IRQ 5
>       Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M]
>       Region 1: I/O ports at 3000 [size=256]
>       Region 2: Memory at fdef0000 (32-bit, non-prefetchable) [size=64K]
>       [virtual] Expansion ROM at fdd00000 [disabled] [size=128K]
>       Capabilities: [50] Power Management version 2
>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> 00: 02 10 5e 51 87 01 90 02 02 00 00 03 10 40 00 00
> 10: 08 00 00 d8 01 30 00 00 00 00 ef fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fb 31
> 30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 08 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fb 31
> 50: 01 00 02 06 00 00 00 00 02 50 20 00 00 02 00 2f
> 60: 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out
> Controller (rev 03)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 22
>       Region 0: I/O ports at 2800 [size=256]
>       Region 1: Memory at fdee0000 (32-bit, non-prefetchable) [size=512]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: hpwdt
>       Kernel modules: hpwdt
> 00: 11 0e 03 b2 03 01 90 02 03 00 80 08 00 00 80 00
> 10: 01 28 00 00 00 00 ee fd 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0a 01 00 00
> 40: 67 fd 00 00 11 00 00 00 00 00 00 00 06 02 06 02
> 50: 00 00 00 00 00 00 00 00 fa 39 fd 77 01 00 00 5a
> 60: 00 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 1a 00 88 03 43 50 51 00
> 90: 30 00 00 00 02 00 00 5f 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 02 00 00 00 00 00 04 04 c1 02
> b0: 00 0a 00 00 00 00 00 00 b7 01 10 51 84 00 03 09
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 01 00 00 00 00 00 00 00 01 00 00 00 00 13 21 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out 
> Processor (rev 03)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64, Cache Line Size: 64 bytes
>       Interrupt: pin B routed to IRQ 23
>       BIST result: 00
>       Region 0: I/O ports at 3400 [size=256]
>       Region 1: Memory at fded0000 (32-bit, non-prefetchable) [size=2K]
>       Region 2: Memory at fdec0000 (32-bit, non-prefetchable) [size=16K]
>       Region 3: Memory at fde00000 (32-bit, non-prefetchable) [size=512K]
>       [virtual] Expansion ROM at fdd20000 [disabled] [size=64K]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: hpilo
>       Kernel modules: hpilo
> 00: 11 0e 04 b2 17 01 90 02 03 00 80 08 10 40 80 80
> 10: 01 34 00 00 00 00 ed fd 00 00 ec fd 00 00 e0 fd
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0b 02 00 00
> 40: 04 0d 00 00 00 00 00 00 00 00 00 00 0a 01 15 12
> 50: 03 e0 ff 00 05 00 40 00 04 00 00 00 00 00 e0 00
> 60: 00 00 00 00 00 00 00 00 8b 60 56 05 59 28 82 f8
> 70: 00 08 00 01 03 03 c0 00 00 00 00 00 00 00 00 00
> 80: bb 03 00 00 57 39 06 00 73 08 4c c5 8e 48 00 00
> 90: 00 00 00 00 03 03 00 00 00 00 00 00 00 00 00 00
> a0: 14 03 02 00 80 00 00 00 20 00 00 00 00 00 00 00
> b0: 10 00 00 00 00 00 e7 81 ff fd 0e 0d 00 00 00 00
> c0: a3 0c 00 00 00 00 07 00 a5 0c 00 00 00 c0 07 00
> d0: 00 00 00 00 00 00 00 00 9a 17 00 00 00 00 00 00
> e0: 00 00 00 00 0b 00 00 00 00 00 00 00 0b 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.4 USB Controller: Hewlett-Packard Company Proliant iLO2 virtual USB
> controller (prog-if 00 [UHCI])
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64
>       Interrupt: pin B routed to IRQ 23
>       Region 4: I/O ports at 3800 [size=32]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: uhci_hcd
>       Kernel modules: uhci-hcd
> 00: 3c 10 00 33 45 01 90 02 00 00 03 0c 00 40 80 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 01 38 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0b 02 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Proliant iLO2 virtual
> UART (prog-if 01)
>       Subsystem: Hewlett-Packard Company Device 3305
>       Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 22
>       Region 0: Memory at fddf0000 (32-bit, non-prefetchable) [size=256]
>       Capabilities: [f0] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Kernel driver in use: ipmi_si
>       Kernel modules: ipmi_si
> 00: 3c 10 02 33 02 00 90 02 00 01 07 0c 00 00 80 00
> 10: 00 00 df fd 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 05 33
> 30: 00 00 00 00 f0 00 00 00 00 00 00 00 0a 01 00 00
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
> 
> 02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev b5)
> (prog-if 00 [Normal decode])
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
>       I/O behind bridge: 0000f000-00000fff
>       Memory behind bridge: fdf00000-fdffffff
>       Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
>       BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>       Capabilities: [60] Express (v1) PCI/PCI-X Bridge, MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1
> <16us
>                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
>               DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
> Unsupported-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> BrConfRtry-
>                       MaxPayload 128 bytes, MaxReadReq 128 bytes
>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency
> L0 <4us, L1 <4us
>                       ClockPM- Suprise- LLActRep- BwNot-
>               LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>       Capabilities: [90] PCI-X bridge device
>               Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD-
> Freq=133MHz
>               Status: Dev=02:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
>               Upstream: Capacity=0 CommitmentLimit=0
>               Downstream: Capacity=0 CommitmentLimit=0
>       Capabilities: [b0] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>       Capabilities: [100] Advanced Error Reporting <?>
>       Capabilities: [14c] Power Budgeting <?>
>       Kernel modules: shpchp
> 00: 66 11 03 01 47 01 10 00 b5 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 02 03 03 40 f1 01 00 22
> 20: f0 fd f0 fd f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 60 00 00 00 00 00 00 00 60 00 03 00
> 40: 00 00 00 00 00 00 00 01 00 00 0c 20 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 c0 00 00 c0
> 60: 10 90 71 00 a0 09 00 00 06 00 10 00 41 6c 01 00
> 70: 00 00 21 10 44 08 00 00 00 00 00 00 00 00 00 00
> 80: 01 00 00 00 20 20 30 2b 92 08 18 20 02 60 00 fb
> 90: 07 b0 c3 00 00 02 00 00 00 00 00 00 00 00 00 00
> a0: 90 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 01 00 c2 c9 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 03:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit
> Ethernet (rev a3)
>       Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server
> Adapter
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64 (16000ns min), Cache Line Size: 64 bytes
>       Interrupt: pin A routed to IRQ 1272
>       Region 0: Memory at fdff0000 (64-bit, non-prefetchable) [size=64K]
>       Region 2: Memory at fdfe0000 (64-bit, non-prefetchable) [size=64K]
>       Expansion ROM at <ignored> [disabled]
>       Capabilities: [40] PCI-X non-bridge device
>               Command: DPERE- ERO- RBC=512 OST=1
>               Status: Dev=03:04.0 64bit+ 133MHz+ SCD- USC- DC=simple
> DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
>       Capabilities: [48] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
>       Capabilities: [50] Vital Product Data <?>
>       Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
> Enable+
>               Address: 00000000fee0300c  Data: 41e1
>       Kernel driver in use: tg3
>       Kernel modules: tg3
> 00: e4 14 78 16 06 00 b0 02 a3 00 00 02 10 40 80 00
> 10: 04 00 ff fd 00 00 00 00 04 00 fe fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 3e 70
> 30: 00 00 e3 82 40 00 00 00 00 00 00 00 0b 01 40 00
> 40: 07 48 00 00 20 03 43 04 01 50 02 c0 00 21 00 64
> 50: 03 58 04 00 3f 61 00 10 05 00 87 00 0c 30 e0 fe
> 60: 00 00 00 00 e1 41 00 00 98 02 03 90 00 80 14 76
> 70: ea 10 00 00 c7 00 00 00 50 00 00 00 00 00 00 00
> 80: 03 58 04 00 d0 00 00 c0 34 00 13 04 82 d0 08 04
> 90: 09 97 00 01 00 00 00 00 00 00 00 00 bf 00 00 00
> a0: 00 00 00 00 f7 01 00 00 00 00 00 00 a2 00 00 00
> b0: 00 00 00 00 00 00 00 65 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> 03:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit
> Ethernet (rev a3)
>       Subsystem: Hewlett-Packard Company NC326i PCIe Dual Port Gigabit Server
> Adapter
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 64 (16000ns min), Cache Line Size: 64 bytes
>       Interrupt: pin B routed to IRQ 17
>       Region 0: Memory at fdfd0000 (64-bit, non-prefetchable) [size=64K]
>       Region 2: Memory at fdfc0000 (64-bit, non-prefetchable) [size=64K]
>       Expansion ROM at <ignored> [disabled]
>       Capabilities: [40] PCI-X non-bridge device
>               Command: DPERE- ERO- RBC=512 OST=1
>               Status: Dev=03:04.1 64bit+ 133MHz+ SCD- USC- DC=simple
> DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
>       Capabilities: [48] Power Management version 2
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 PME-Enable- DSel=0 DScale=1 PME-
>       Capabilities: [50] Vital Product Data <?>
>       Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
> Enable-
>               Address: 00000000fee0300c  Data: 41c9
>       Kernel driver in use: tg3
>       Kernel modules: tg3
> 00: e4 14 78 16 46 01 b0 02 a3 00 00 02 10 40 80 00
> 10: 04 00 fd fd 00 00 00 00 04 00 fc fd 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 3e 70
> 30: 00 00 6f ba 40 00 00 00 00 00 00 00 0a 02 40 00
> 40: 07 48 00 00 21 03 43 04 01 50 02 c0 00 20 00 64
> 50: 03 58 e8 00 6b ee 9b 7d 05 00 86 00 0c 30 e0 fe
> 60: 00 00 00 00 c9 41 00 00 9a 02 03 90 00 80 14 76
> 70: ea 10 00 00 c7 00 00 00 14 04 00 00 00 00 00 00
> 80: 29 0a 99 5a 2a 00 00 00 34 00 00 00 fe d0 08 04
> 90: 01 07 00 09 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>

Comment 8 Bernhard Schmidt 2009-03-19 11:07:56 UTC

On Thu, Mar 19, 2009 at 09:58:42AM -0700, Matt Carlson wrote:

Hello Matt,

> Hi Bernhard.  I talked to Michael about this and we'd like you to try
> two things.
> 
> 1) Can you disable iLo2 and see if you can still reproduce the problem?

That will take a few days, I'll ask the on-site guys to check whether we
have an external IP-KVM available. 

Would switching the uplink connection to eth1 (and not use eth0 in
Linux) give some more clue as well?

> 2) Can you apply the following patch to get more information on when
>    MMIO gets disabled?

Applied, I'll send you the information as soon as it happens again
(which seems to be happening rather often the last couple of days).

Bernhard

Comment 9 Matt Carlson 2009-03-19 11:17:35 UTC

On Thu, Mar 19, 2009 at 11:06:44AM -0700, Bernhard Schmidt wrote:
> On Thu, Mar 19, 2009 at 09:58:42AM -0700, Matt Carlson wrote:
> 
> Hello Matt,
> 
> > Hi Bernhard.  I talked to Michael about this and we'd like you to try
> > two things.
> > 
> > 1) Can you disable iLo2 and see if you can still reproduce the problem?
> 
> That will take a few days, I'll ask the on-site guys to check whether we
> have an external IP-KVM available. 
> 
> Would switching the uplink connection to eth1 (and not use eth0 in
> Linux) give some more clue as well?

Sure, let's try that.  Maybe this is the better way to go anyways.  I just
learned that disabling iLo2 doesn't necessarily disable the management
firmware on the network device.

For this to be a meaningful test though, we need to verify that the
driver sign-on messages have a line that reads "ASF[0]" on eth1.  Can
you confirm?

> > 2) Can you apply the following patch to get more information on when
> >    MMIO gets disabled?
> 
> Applied, I'll send you the information as soon as it happens again
> (which seems to be happening rather often the last couple of days).

Thanks.

Comment 10 Bernhard Schmidt 2009-03-19 11:20:22 UTC

On Thu, Mar 19, 2009 at 11:15:36AM -0700, Matt Carlson wrote:

> > Would switching the uplink connection to eth1 (and not use eth0 in
> > Linux) give some more clue as well?
> Sure, let's try that.  Maybe this is the better way to go anyways.  I just
> learned that disabling iLo2 doesn't necessarily disable the management
> firmware on the network device.
> 
> For this to be a meaningful test though, we need to verify that the
> driver sign-on messages have a line that reads "ASF[0]" on eth1.  Can
> you confirm?

Yes:

[1186598.218205] eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1]
WireSpeed[1] TSOcap[1]
[1186598.281314] eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0]
WireSpeed[1] TSOcap[1]

I'll keep the traffic on eth0 now until it breaks again, hopefully
spewing out more debugging info with your patch and then I'll switch to
eth1.

Bernhard

Comment 11 Bernhard Schmidt 2009-03-22 06:22:11 UTC

On Thu, Mar 19, 2009 at 09:58:42AM -0700, Matt Carlson wrote:

Hello Matt,

> Hi Bernhard.  I talked to Michael about this and we'd like you to try
> two things.
> 
> 1) Can you disable iLo2 and see if you can still reproduce the problem?
> 2) Can you apply the following patch to get more information on when
>    MMIO gets disabled?

Not sure this is going to help you. NIC crashed two times tonight, logs
look like this

Mar 22 04:06:46 svr02 kernel: [1392136.468921] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:47 svr02 kernel: [1392137.520288] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:48 svr02 kernel: [1392138.568267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:49 svr02 kernel: [1392139.616266] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:50 svr02 kernel: [1392140.664266] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:51 svr02 kernel: [1392141.712276] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:52 svr02 kernel: [1392142.760297] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:53 svr02 kernel: [1392143.808258] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:54 svr02 kernel: [1392144.856256] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:55 svr02 kernel: [1392145.904266] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:56 svr02 kernel: [1392146.952276] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:57 svr02 kernel: [1392148.000267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:58 svr02 kernel: [1392149.048271] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:06:59 svr02 kernel: [1392150.096268] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:00 svr02 kernel: [1392151.144277] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:01 svr02 kernel: [1392152.192287] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:02 svr02 kernel: [1392153.240268] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:03 svr02 kernel: [1392154.288267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:04 svr02 kernel: [1392155.336267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:05 svr02 kernel: [1392156.384268] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:07 svr02 kernel: [1392157.432267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:08 svr02 kernel: [1392158.480266] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:09 svr02 kernel: [1392159.528289] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:10 svr02 kernel: [1392160.576297] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:11 svr02 kernel: [1392161.624277] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:12 svr02 kernel: [1392162.672267] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:13 svr02 kernel: [1392163.720266] PCI Memory Mapped IO Disabled!!!!
Mar 22 04:07:14 svr02 kernel: [1392164.768266] PCI Memory Mapped IO Disabled!!!!

at this point the "watchdog" kicked in and did rmmod/modprobe, so I
think the only thing you can read out of this debugging log is that
there was no kernel message right before MMIO got disabled and it takes
quite a while to fire the Tx timeout.

Mar 22 04:07:15 svr02 kernel: [1392165.540078] tg3 0000:03:04.1: PCI INT B disabled
Mar 22 04:07:16 svr02 kernel: [1392166.817125] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
Mar 22 04:07:18 svr02 kernel: [1392168.398844] tg3: eth0: No firmware running.
Mar 22 04:07:29 svr02 kernel: [1392179.793309] tg3: eth0: Link is down.
Mar 22 04:07:31 svr02 kernel: [1392181.896030] tg3 0000:03:04.0: PCI INT A disabled
Mar 22 04:07:33 svr02 kernel: [1392183.957132] tg3.c:v3.94 (August 14, 2008)
Mar 22 04:07:33 svr02 kernel: [1392184.020034] tg3 0000:03:04.0: enabling device (0000 -> 0002)
Mar 22 04:07:33 svr02 kernel: [1392184.086083] tg3 0000:03:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16

I'm now switching to eth1.

Bernhard

Comment 12 Matt Carlson 2009-03-23 11:20:28 UTC

On Sun, Mar 22, 2009 at 06:21:21AM -0700, Bernhard Schmidt wrote:
> On Thu, Mar 19, 2009 at 09:58:42AM -0700, Matt Carlson wrote:
> 
> Hello Matt,
> 
> > Hi Bernhard.  I talked to Michael about this and we'd like you to try
> > two things.
> > 
> > 1) Can you disable iLo2 and see if you can still reproduce the problem?
> > 2) Can you apply the following patch to get more information on when
> >    MMIO gets disabled?
> 
> Not sure this is going to help you. NIC crashed two times tonight, logs
> look like this

Actually, it does. :)

> Mar 22 04:06:46 svr02 kernel: [1392136.468921] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:47 svr02 kernel: [1392137.520288] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:48 svr02 kernel: [1392138.568267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:49 svr02 kernel: [1392139.616266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:50 svr02 kernel: [1392140.664266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:51 svr02 kernel: [1392141.712276] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:52 svr02 kernel: [1392142.760297] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:53 svr02 kernel: [1392143.808258] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:54 svr02 kernel: [1392144.856256] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:55 svr02 kernel: [1392145.904266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:56 svr02 kernel: [1392146.952276] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:57 svr02 kernel: [1392148.000267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:58 svr02 kernel: [1392149.048271] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:59 svr02 kernel: [1392150.096268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:00 svr02 kernel: [1392151.144277] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:01 svr02 kernel: [1392152.192287] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:02 svr02 kernel: [1392153.240268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:03 svr02 kernel: [1392154.288267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:04 svr02 kernel: [1392155.336267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:05 svr02 kernel: [1392156.384268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:07 svr02 kernel: [1392157.432267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:08 svr02 kernel: [1392158.480266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:09 svr02 kernel: [1392159.528289] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:10 svr02 kernel: [1392160.576297] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:11 svr02 kernel: [1392161.624277] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:12 svr02 kernel: [1392162.672267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:13 svr02 kernel: [1392163.720266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:14 svr02 kernel: [1392164.768266] PCI Memory Mapped IO
> Disabled!!!!
> 
> at this point the "watchdog" kicked in and did rmmod/modprobe, so I
> think the only thing you can read out of this debugging log is that
> there was no kernel message right before MMIO got disabled and it takes
> quite a while to fire the Tx timeout.

So traffic on this box must be pretty light for the watchdog to fire off
30 seconds after the MMIO problem was detected, right?  Interesting.

> Mar 22 04:07:15 svr02 kernel: [1392165.540078] tg3 0000:03:04.1: PCI INT B
> disabled
> Mar 22 04:07:16 svr02 kernel: [1392166.817125] tg3: tg3_abort_hw timed out
> for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
> Mar 22 04:07:18 svr02 kernel: [1392168.398844] tg3: eth0: No firmware
> running.
> Mar 22 04:07:29 svr02 kernel: [1392179.793309] tg3: eth0: Link is down.
> Mar 22 04:07:31 svr02 kernel: [1392181.896030] tg3 0000:03:04.0: PCI INT A
> disabled
> Mar 22 04:07:33 svr02 kernel: [1392183.957132] tg3.c:v3.94 (August 14, 2008)
> Mar 22 04:07:33 svr02 kernel: [1392184.020034] tg3 0000:03:04.0: enabling
> device (0000 -> 0002)
> Mar 22 04:07:33 svr02 kernel: [1392184.086083] tg3 0000:03:04.0: PCI INT A ->
> GSI 16 (level, low) -> IRQ 16
> 
> I'm now switching to eth1.

O.K.  I eagerly await your results.

Comment 13 Bernhard Schmidt 2009-03-23 17:36:59 UTC

On 23.03.2009 19:18, Matt Carlson wrote:

Hello Matt,

>> Mar 22 04:06:46 svr02 kernel: [1392136.468921] PCI Memory Mapped IO
>> Disabled!!!!
[...]
>> Mar 22 04:07:14 svr02 kernel: [1392164.768266] PCI Memory Mapped IO
>> Disabled!!!!
>> at this point the "watchdog" kicked in and did rmmod/modprobe, so I
>> think the only thing you can read out of this debugging log is that
>> there was no kernel message right before MMIO got disabled and it takes
>> quite a while to fire the Tx timeout.
> So traffic on this box must be pretty light for the watchdog to fire off
> 30 seconds after the MMIO problem was detected, right?  Interesting.

Just to make sure I didn't confuse you, the "watchdog" I was talking 
about here is a shellscript like this, executed every minute

---
/bin/ping -q -c 5 <defaultgw> > /dev/null
RC=$?
if [ ${RC} -ne 0 ]; then
	rmmod tg3; sleep 5; modprobe tg3; sleep 5; ifup --force eth0
fi
---

at :46 MMIO was disabled, at :00 the cronjob started which took until 
:15 before detecting the network was dead and reloaded the modules

>> Mar 22 04:07:15 svr02 kernel: [1392165.540078] tg3 0000:03:04.1: PCI INT B
>> disabled
>> Mar 22 04:07:16 svr02 kernel: [1392166.817125] tg3: tg3_abort_hw timed out
>> for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
>> Mar 22 04:07:18 svr02 kernel: [1392168.398844] tg3: eth0: No firmware
>> running.
>> Mar 22 04:07:29 svr02 kernel: [1392179.793309] tg3: eth0: Link is down.
>> Mar 22 04:07:31 svr02 kernel: [1392181.896030] tg3 0000:03:04.0: PCI INT A
>> disabled
>> Mar 22 04:07:33 svr02 kernel: [1392183.957132] tg3.c:v3.94 (August 14, 2008)
>> Mar 22 04:07:33 svr02 kernel: [1392184.020034] tg3 0000:03:04.0: enabling
>> device (0000 ->  0002)
>> Mar 22 04:07:33 svr02 kernel: [1392184.086083] tg3 0000:03:04.0: PCI INT A
>> ->  GSI 16 (level, low) ->  IRQ 16

The tg3 watchdog (tg3: eth0: transmit timed out, resetting) did not 
appear at all in this circle, so I guess the checkscript killed the 
module before.

Yes, the NIC is very lightly loaded, around 100kbps / 70pps in each 
direction with a few occasional spikes.

>> I'm now switching to eth1.
> O.K.  I eagerly await your results.

So far so good, but it has only been running ~36 hours, that's not 
really a stability spree yet :-)

I'll keep you updated.

Bernhard

Comment 14 Matt Carlson 2009-03-31 16:26:31 UTC

On Sun, Mar 22, 2009 at 06:21:21AM -0700, Bernhard Schmidt wrote:
> On Thu, Mar 19, 2009 at 09:58:42AM -0700, Matt Carlson wrote:
> 
> Hello Matt,
> 
> > Hi Bernhard.  I talked to Michael about this and we'd like you to try
> > two things.
> > 
> > 1) Can you disable iLo2 and see if you can still reproduce the problem?
> > 2) Can you apply the following patch to get more information on when
> >    MMIO gets disabled?
> 
> Not sure this is going to help you. NIC crashed two times tonight, logs
> look like this
> 
> Mar 22 04:06:46 svr02 kernel: [1392136.468921] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:47 svr02 kernel: [1392137.520288] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:48 svr02 kernel: [1392138.568267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:49 svr02 kernel: [1392139.616266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:50 svr02 kernel: [1392140.664266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:51 svr02 kernel: [1392141.712276] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:52 svr02 kernel: [1392142.760297] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:53 svr02 kernel: [1392143.808258] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:54 svr02 kernel: [1392144.856256] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:55 svr02 kernel: [1392145.904266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:56 svr02 kernel: [1392146.952276] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:57 svr02 kernel: [1392148.000267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:58 svr02 kernel: [1392149.048271] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:06:59 svr02 kernel: [1392150.096268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:00 svr02 kernel: [1392151.144277] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:01 svr02 kernel: [1392152.192287] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:02 svr02 kernel: [1392153.240268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:03 svr02 kernel: [1392154.288267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:04 svr02 kernel: [1392155.336267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:05 svr02 kernel: [1392156.384268] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:07 svr02 kernel: [1392157.432267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:08 svr02 kernel: [1392158.480266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:09 svr02 kernel: [1392159.528289] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:10 svr02 kernel: [1392160.576297] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:11 svr02 kernel: [1392161.624277] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:12 svr02 kernel: [1392162.672267] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:13 svr02 kernel: [1392163.720266] PCI Memory Mapped IO
> Disabled!!!!
> Mar 22 04:07:14 svr02 kernel: [1392164.768266] PCI Memory Mapped IO
> Disabled!!!!
> 
> at this point the "watchdog" kicked in and did rmmod/modprobe, so I
> think the only thing you can read out of this debugging log is that
> there was no kernel message right before MMIO got disabled and it takes
> quite a while to fire the Tx timeout.
> 
> Mar 22 04:07:15 svr02 kernel: [1392165.540078] tg3 0000:03:04.1: PCI INT B
> disabled
> Mar 22 04:07:16 svr02 kernel: [1392166.817125] tg3: tg3_abort_hw timed out
> for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
> Mar 22 04:07:18 svr02 kernel: [1392168.398844] tg3: eth0: No firmware
> running.
> Mar 22 04:07:29 svr02 kernel: [1392179.793309] tg3: eth0: Link is down.
> Mar 22 04:07:31 svr02 kernel: [1392181.896030] tg3 0000:03:04.0: PCI INT A
> disabled
> Mar 22 04:07:33 svr02 kernel: [1392183.957132] tg3.c:v3.94 (August 14, 2008)
> Mar 22 04:07:33 svr02 kernel: [1392184.020034] tg3 0000:03:04.0: enabling
> device (0000 -> 0002)
> Mar 22 04:07:33 svr02 kernel: [1392184.086083] tg3 0000:03:04.0: PCI INT A ->
> GSI 16 (level, low) -> IRQ 16
> 
> I'm now switching to eth1.

Bernhard, any word on what happened?

Comment 15 Bernhard Schmidt 2009-03-31 22:16:28 UTC

On Tue, Mar 31, 2009 at 09:26:17AM -0700, Matt Carlson wrote:

> > I'm now switching to eth1.
> Bernhard, any word on what happened?

So far so good. In the last week my watchdog (cannot reach the default
gateway) triggered once, but since there were no "PCI Memory Mapped IO
Disabled!!!!" messages in the logfile I assume that was a real network
problem. Since the box ran fine for months initially and the first two
occurances of this issue were two weeks apart I cannot say for sure, but
it definitely feels better than the "once in two days" in the end.

Bernhard

Comment 16 Bernhard Schmidt 2009-04-13 21:54:45 UTC

Hi Matt,

>>> I'm now switching to eth1.
>> Bernhard, any word on what happened?
> 
> So far so good. In the last week my watchdog (cannot reach the default
> gateway) triggered once, but since there were no "PCI Memory Mapped IO
> Disabled!!!!" messages in the logfile I assume that was a real network
> problem. Since the box ran fine for months initially and the first two
> occurances of this issue were two weeks apart I cannot say for sure, but
> it definitely feels better than the "once in two days" in the end.

No crashes in the last two weeks.

Do you have any further suggestions how to debug this or should we 
accept that portsharing doesn't work very well? This is the second 
problem we can directly attribute to the sharing with the iLO (the first 
one being the "no IPv6 unless the port is in promiscous mode", we had a 
thread about this here on netdev a few months ago).

Bernhard

Comment 17 Matt Carlson 2009-04-14 18:29:34 UTC

On Mon, Apr 13, 2009 at 02:54:40PM -0700, Bernhard Schmidt wrote:
> Hi Matt,
> 
> >>> I'm now switching to eth1.
> >> Bernhard, any word on what happened?
> > 
> > So far so good. In the last week my watchdog (cannot reach the default
> > gateway) triggered once, but since there were no "PCI Memory Mapped IO
> > Disabled!!!!" messages in the logfile I assume that was a real network
> > problem. Since the box ran fine for months initially and the first two
> > occurances of this issue were two weeks apart I cannot say for sure, but
> > it definitely feels better than the "once in two days" in the end.
> 
> No crashes in the last two weeks.
> 
> Do you have any further suggestions how to debug this or should we 
> accept that portsharing doesn't work very well? This is the second 
> problem we can directly attribute to the sharing with the iLO (the first 
> one being the "no IPv6 unless the port is in promiscous mode", we had a 
> thread about this here on netdev a few months ago).

No, I think we need to get this to work with portsharing.  We just need
to figure out what it is about it that causes these types of errors.

I talked to the firmware maintainer here.  We have a couple ideas that
might uncover what is happening.  Our next step is to develop a set of
tests that will show under what assumptions the firmware is operating.
Once we have that, I'll ask you to patch your driver so that we can see
what is happening from your end.

Stay tuned.

Comment 18 goran 2010-02-17 06:27:38 UTC

Hi all,
I'm sorry if I write in wrong bug report but it seams identical like mine.

Bug:
http://www.kerneloops.org/submitresult.php?number=2554316

WARNING: at /tmp/buildd/linux-2.6-2.6.32/debian/build/source_i386_none/net/sched/sch_generic.c:261 dev_watchdog+0xbd/0x15d()
Hardware name: 40682VG         
NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Modules linked in: uinput ppdev lp parport sco bridge stp bnep rfcomm l2cap crc16 vboxnetadp vboxnetflt vboxdrv acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_conservative cpufreq_userspace microcode fuse snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer uvcvideo btusb videodev v4l1_compat snd_seq_device lib80211_crypt_tkip bluetooth joydev rfkill i2c_i801 snd wl(P) soundcore snd_page_alloc rng_core serio_raw lib80211 evdev pcspkr ac psmouse processor battery ext3 jbd mbcache sd_mod crc_t10dif ide_gd_mod ata_generic usbhid hid usb_storage ide_pci_generic i915 drm_kms_helper drm i2c_algo_bit ata_piix uhci_hcd i2c_core intel_agp piix ehci_hcd libata tg3 video output ide_core usbcore agpgart scsi_mod libphy button nls_base thermal fan thermal_sys [last unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Tainted: P           2.6.32-trunk-686 #1
Call Trace:
 [<c1030ba1>] ? warn_slowpath_common+0x5e/0x8a
 [<c11d778c>] ? dev_watchdog+0x0/0x15d
 [<c1030bff>] ? warn_slowpath_fmt+0x26/0x2a
 [<c11d7849>] ? dev_watchdog+0xbd/0x15d
 [<c102520b>] ? __wake_up+0x29/0x39
 [<c1041826>] ? insert_work+0x71/0x78
 [<c1041b9f>] ? delayed_work_timer_fn+0x0/0x28
 [<c103b360>] ? run_timer_softirq+0x16a/0x1eb
 [<c1035c80>] ? __do_softirq+0xaa/0x151
 [<c1035d58>] ? do_softirq+0x31/0x3c
 [<c1035e2e>] ? irq_exit+0x26/0x58
 [<c1004c91>] ? do_IRQ+0x78/0x89
 [<c10037f0>] ? common_interrupt+0x30/0x38
 [<f7f7a0d7>] ? acpi_idle_enter_bm+0x246/0x281 [processor]
 [<c11b5421>] ? cpuidle_idle_call+0x68/0xbb
 [<c1002388>] ? cpu_idle+0x89/0xa5
 [<c1399800>] ? start_kernel+0x30c/0x311
---[ end trace 0e630f751e4c6fe2 ]---

I have Lenovo IdeaPad S10e (Broadcom NIC)

lspci -vnn
02:00.0 Ethernet controller [0200]: Broadcom Corporation NetLink BCM5906M Fast Ethernet PCI Express [14e4:1713] (rev 02)
        Subsystem: Lenovo IdeaPad S10e [17aa:3a23]
        Flags: bus master, fast devsel, latency 0, IRQ 27
        Memory at f0200000 (64-bit, non-prefetchable) [size=64K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: <access denied>
        Kernel driver in use: tg3

Problem is when I upload bigger data on eth0, ethernet broke after 10-15 minutes and if I don't restart him with /etc/init.d/networking stop | start he doesn't come back for ~3 minutes.
There is no problem with download.

I'm sure you have doesn't of mine kerneloops report like this.

Thanks.

Comment 19 Richard Wall 2010-04-23 16:20:47 UTC

There seem to be many similar bugs already listed - this one seemed most similar to mine. We have a pair of very busy Squid servers using identical Asus server motherboards. The networking on one box intermittently fails. Here's the resulting syslog / dmesg

# cat /sys/class/net/eth0/device/vendor 
0x14e4
# cat /sys/class/net/eth0/device/device 
0x1659

# uname -r
2.6.31.12

#cat /var/log/syslog
Apr 22 23:49:35  kernel: [623449.988504] ------------[ cut here ]------------
Apr 22 23:49:35  kernel: [623449.988511] WARNING: at net/sched/sch_generic.c:246 dev_watchdog+0x1be/0x1d0()
Apr 22 23:49:35  kernel: [623449.988514] Hardware name: System Product Name
Apr 22 23:49:35  kernel: [623449.988516] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Apr 22 23:49:35  kernel: [623449.988518] Modules linked in: ip_gre e1000 via_rhine tg3 libphy r8169 pcnet32 e100 8139too mii w83627ehf vt8231 via686a hwm
on_vid coretemp asus_atk0110 hwmon
Apr 22 23:49:35  kernel: [623449.988533] Pid: 0, comm: swapper Not tainted 2.6.31.12 #2
Apr 22 23:49:35  kernel: [623449.988535] Call Trace:
Apr 22 23:49:35  kernel: [623449.988540]  [<c012864e>] ? warn_slowpath_common+0x6e/0xb0
Apr 22 23:49:35  kernel: [623449.988543]  [<c034e4fe>] ? dev_watchdog+0x1be/0x1d0
Apr 22 23:49:35  kernel: [623449.988546]  [<c01286db>] ? warn_slowpath_fmt+0x2b/0x30
Apr 22 23:49:35  kernel: [623449.988549]  [<c034e4fe>] ? dev_watchdog+0x1be/0x1d0
Apr 22 23:49:35  kernel: [623449.988553]  [<c011f842>] ? __wake_up+0x42/0x60
Apr 22 23:49:35  kernel: [623449.988557]  [<c01376f2>] ? insert_work+0x42/0x50
Apr 22 23:49:35  kernel: [623449.988560]  [<c034e340>] ? dev_watchdog+0x0/0x1d0
Apr 22 23:49:35  kernel: [623449.988564]  [<c0131149>] ? run_timer_softirq+0xf9/0x1c0
Apr 22 23:49:35  kernel: [623449.988567]  [<c012d2c0>] ? __do_softirq+0x80/0x100
Apr 22 23:49:35  kernel: [623449.988570]  [<c012d36d>] ? do_softirq+0x2d/0x40
Apr 22 23:49:35  kernel: [623449.988574]  [<c0114d94>] ? smp_apic_timer_interrupt+0x54/0x90
Apr 22 23:49:35  kernel: [623449.988577]  [<c0103676>] ? apic_timer_interrupt+0x2a/0x30
Apr 22 23:49:35  kernel: [623449.988581]  [<c03f00d8>] ? klist_add_before+0x18/0x50
Apr 22 23:49:35  kernel: [623449.988585]  [<c0109dc2>] ? mwait_idle+0x42/0x60
Apr 22 23:49:35  kernel: [623449.988587]  [<c0101d55>] ? cpu_idle+0x35/0x60
Apr 22 23:49:35  kernel: [623449.988590] ---[ end trace 346a74434bf31555 ]---
Apr 22 23:49:35  kernel: [623449.988592] tg3: eth0: transmit timed out, resetting
Apr 22 23:49:35  kernel: [623449.988596] tg3: DEBUG: MAC_TX_STATUS[0000000f] MAC_RX_STATUS[00000008]
Apr 22 23:49:35  kernel: [623449.988601] tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Apr 22 23:49:35  kernel: [623450.089694] tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
Apr 22 23:49:35  kernel: [623450.248154] tg3: eth0: Link is down.


# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
        Link detected: no

Comment 20 Alan 2012-10-30 16:42:01 UTC

If this is still seen on modern kernels then please re-open/update

Comment 21 Angel Genchev 2014-10-30 13:15:17 UTC

(In reply to Alan from comment #20)
> If this is still seen on modern kernels then please re-open/update

Yes, kernel 3.13.xxx on x86_64 (HP server DL 385 G8, 4-port broadcom-based nic)
Oct 30 14:28:27 Obi-One kernel: [12744.969716] ------------[ cut here ]------------
Oct 30 14:28:27 Obi-One kernel: [12744.969739] WARNING: CPU: 0 PID: 0 at /build/buildd/linux-3.13.0/net/sched/sch_generic.c:264 dev_watchdog+0x276/0x280()
Oct 30 14:28:27 Obi-One kernel: [12744.969744] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Oct 30 14:28:27 Obi-One kernel: [12744.969746] Modules linked in: vhost_net vhost macvtap macvlan xt_recent ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables bridge stp llc kvm_amd kvm crct10dif_pclmul dm_multipath crc32_pclmul ghash_clmulni_intel aesni_intel scsi_dh aes_x86_64 lrw gf128mul glue_helper sp5100_tco amd64_edac_mod ablk_helper edac_core i2c_piix4 ipmi_si hpwdt mac_hid hpilo cryptd acpi_power_meter k10temp fam15h_power serio_raw edac_mce_amd parport_pc ppdev lp parport pata_acpi hid_generic psmouse tg3 usbhid hid ahci ptp usb_storage pata_atiixp libahci hpsa pps_core
Oct 30 14:28:27 Obi-One kernel: [12744.969868] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-39-generic #66-Ubuntu
Oct 30 14:28:27 Obi-One kernel: [12744.969872] Hardware name: HP ProLiant DL385p Gen8, BIOS A28 11/02/2013
Oct 30 14:28:27 Obi-One kernel: [12744.969885]  0000000000000009 ffff880237c03d98 ffffffff8171ece7 ffff880237c03de0
Oct 30 14:28:27 Obi-One kernel: [12744.969911]  ffff880237c03dd0 ffffffff8106773d 0000000000000000 ffff880831c90000
Oct 30 14:28:27 Obi-One kernel: [12744.969940]  ffff880832330d80 0000000000000005 0000000000000000 ffff880237c03e30
Oct 30 14:28:27 Obi-One kernel: [12744.969957] Call Trace:
Oct 30 14:28:27 Obi-One kernel: [12744.969966]  <IRQ>  [<ffffffff8171ece7>] dump_stack+0x45/0x56
Oct 30 14:28:27 Obi-One kernel: [12744.969993]  [<ffffffff8106773d>] warn_slowpath_common+0x7d/0xa0
Oct 30 14:28:27 Obi-One kernel: [12744.970006]  [<ffffffff810677ac>] warn_slowpath_fmt+0x4c/0x50
Oct 30 14:28:27 Obi-One kernel: [12744.970020]  [<ffffffff816438f6>] dev_watchdog+0x276/0x280
Oct 30 14:28:27 Obi-One kernel: [12744.970032]  [<ffffffff81643680>] ? dev_graft_qdisc+0x80/0x80
Oct 30 14:28:27 Obi-One kernel: [12744.970045]  [<ffffffff810742f6>] call_timer_fn+0x36/0x100
Oct 30 14:28:27 Obi-One kernel: [12744.970054]  [<ffffffff81643680>] ? dev_graft_qdisc+0x80/0x80
Oct 30 14:28:27 Obi-One kernel: [12744.970066]  [<ffffffff8107528f>] run_timer_softirq+0x1ef/0x2f0
Oct 30 14:28:27 Obi-One kernel: [12744.970087]  [<ffffffff8106cb8c>] __do_softirq+0xec/0x2c0
Oct 30 14:28:27 Obi-One kernel: [12744.970100]  [<ffffffff8106d0d5>] irq_exit+0x105/0x110
Oct 30 14:28:27 Obi-One kernel: [12744.970112]  [<ffffffff81731bc5>] smp_apic_timer_interrupt+0x45/0x60
Oct 30 14:28:27 Obi-One kernel: [12744.970127]  [<ffffffff8173055d>] apic_timer_interrupt+0x6d/0x80
Oct 30 14:28:27 Obi-One kernel: [12744.970135]  <EOI>  [<ffffffff815d189f>] ? cpuidle_enter_state+0x4f/0xc0
Oct 30 14:28:27 Obi-One kernel: [12744.970161]  [<ffffffff815d19c9>] cpuidle_idle_call+0xb9/0x1f0
Oct 30 14:28:27 Obi-One kernel: [12744.970176]  [<ffffffff8101d1de>] arch_cpu_idle+0xe/0x30
Oct 30 14:28:27 Obi-One kernel: [12744.970188]  [<ffffffff810bed85>] cpu_startup_entry+0xc5/0x290
Oct 30 14:28:27 Obi-One kernel: [12744.970204]  [<ffffffff8170d1f7>] rest_init+0x77/0x80
Oct 30 14:28:27 Obi-One kernel: [12744.970218]  [<ffffffff81d36f70>] start_kernel+0x438/0x443
Oct 30 14:28:27 Obi-One kernel: [12744.970229]  [<ffffffff81d36941>] ? repair_env_string+0x5c/0x5c
Oct 30 14:28:27 Obi-One kernel: [12744.970245]  [<ffffffff81d36120>] ? early_idt_handlers+0x120/0x120
Oct 30 14:28:27 Obi-One kernel: [12744.970257]  [<ffffffff81d365ee>] x86_64_start_reservations+0x2a/0x2c
Oct 30 14:28:27 Obi-One kernel: [12744.970264]  [<ffffffff81d36733>] x86_64_start_kernel+0x143/0x152
Oct 30 14:28:27 Obi-One kernel: [12744.970268] ---[ end trace dc4b8daef1a66ac0 ]---
Oct 30 14:28:27 Obi-One kernel: [12744.970275] tg3 0000:04:00.0 eth0: transmit timed out, resetting
Oct 30 14:28:28 Obi-One kernel: [12746.281243] tg3 0000:04:00.0 eth0: 0x00000000: 0x165714e4, 0x00100446, 0x02000001, 0x00800010
Oct 30 14:28:28 Obi-One kernel: [12746.281255] tg3 0000:04:00.0 eth0: 0x00000010: 0xfcbf000c, 0x00000000, 0xfcbe000c, 0x00000000
Oct 30 14:28:28 Obi-One kernel: [12746.281258] tg3 0000:04:00.0 eth0: 0x00000020: 0xfcbd000c, 0x00000000, 0x00000000, 0x169d103c
Oct 30 14:28:28 Obi-One kernel: [12746.281260] tg3 0000:04:00.0 eth0: ....... follows several screens hex dump.....

 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
04:00.0 0200: 14e4:1657 (rev 01)
	Subsystem: 103c:169d
	Flags: bus master, fast devsel, latency 0, IRQ 36
	Memory at fcbf0000 (64-bit, prefetchable) [size=64K]
	Memory at fcbe0000 (64-bit, prefetchable) [size=64K]
	Memory at fcbd0000 (64-bit, prefetchable) [size=64K]
	[virtual] Expansion ROM at fcb00000 [disabled] [size=128K]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable+ Count=17 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [13c] Device Serial Number 00-00-ac-16-2d-72-98-bc
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Capabilities: [230] Transaction Processing Hints
	Kernel driver in use: tg3

Comment 22 Angel Genchev 2014-10-30 13:30:20 UTC

rmmod/modprobe tg3 does not help. To restore the connectivity, I have to pull the plug. The connectivity on the other ports is O.K. 
I don't know if related, but the error is easily triggered when a qemu/KVM guest tries to send a packet via the NIC. There are some unhandled rdmsr/wrmsr warnings from kvm just before the call trace:
Oct 30 14:26:20 Obi-One kernel: [12618.383147] device vnet1 entered promiscuous mode
Oct 30 14:26:20 Obi-One kernel: [12618.408736] virbr0: port 1(vnet1) entered listening state
Oct 30 14:26:20 Obi-One kernel: [12618.408760] virbr0: port 1(vnet1) entered listening state
Oct 30 14:26:22 Obi-One kernel: [12620.129653] kvm [5194]: vcpu0 unhandled rdmsr: 0xc0011021
Oct 30 14:26:22 Obi-One kernel: [12620.129673] kvm [5194]: vcpu0 unhandled rdmsr: 0xc0010112
Oct 30 14:26:22 Obi-One kernel: [12620.292889] kvm [5194]: vcpu0 unhandled rdmsr: 0xc0010001
Oct 30 14:26:22 Obi-One kernel: [12620.306673] kvm [5194]: vcpu1 unhandled rdmsr: 0xc0011021
Oct 30 14:26:22 Obi-One kernel: [12620.409849] virbr0: port 1(vnet1) entered learning state
Oct 30 14:26:24 Obi-One kernel: [12622.414480] virbr0: topology change detected, propagating
Oct 30 14:26:24 Obi-One kernel: [12622.414493] virbr0: port 1(vnet1) entered forwarding state
Oct 30 14:26:24 Obi-One kernel: [12622.414555] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready
Oct 30 14:26:25 Obi-One kernel: [12623.137360] kvm [5194]: vcpu0 unhandled rdmsr: 0x3a

Comment 23 phodamentals 2014-11-09 16:57:07 UTC

I am encountering very similar issues as well with kernel 2.6.32-431.29.2 (the latest for CentOS 6.5 at the time of this writing.) and the Tigon NIC with driver 3.136H that my vendor provided me. On average, it occurs every 25 days and requires a reboot of the server. Older drivers yield more frequent unresponsiveness spells. This happens on all the servers in our Hadoop cluster.
(Broadcom Corporation NetXtreme BCM5725 Gigabit Ethernet PCIe (rev 10))

--------------------------------------

Nov  8 04:32:00 whale12 kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted)
Nov  8 04:32:00 whale12 kernel: Hardware name: Altus OCP31
Nov  8 04:32:00 whale12 kernel: NETDEV WATCHDOG: eth2 (tg3): transmit queue 0 timed out
Nov  8 04:32:00 whale12 kernel: Modules linked in: iptable_filter ip_tables tg3(U) ptp pps_core autofs4 cpufreq_ondemand powernow_k8 freq_table mperf ipv6 ext3 jbd uinput microcode sg fam15h_power k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: pps_core]
Nov  8 04:32:00 whale12 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.29.2.el6.x86_64 #1
Nov  8 04:32:00 whale12 kernel: Call Trace:
Nov  8 04:32:00 whale12 kernel: <IRQ>  [<ffffffff81071b37>] ? warn_slowpath_common+0x87/0xc0
Nov  8 04:32:00 whale12 kernel: [<ffffffff81071c26>] ? warn_slowpath_fmt+0x46/0x50
Nov  8 04:32:00 whale12 kernel: [<ffffffff8147c75b>] ? dev_watchdog+0x26b/0x280
Nov  8 04:32:00 whale12 kernel: [<ffffffff81094ccd>] ? insert_work+0x6d/0xb0
Nov  8 04:32:00 whale12 kernel: [<ffffffff8147c4f0>] ? dev_watchdog+0x0/0x280
Nov  8 04:32:00 whale12 kernel: [<ffffffff810847f7>] ? run_timer_softirq+0x197/0x340
Nov  8 04:32:00 whale12 kernel: [<ffffffff810ac585>] ? tick_dev_program_event+0x65/0xc0
Nov  8 04:32:00 whale12 kernel: [<ffffffff8107a5f1>] ? __do_softirq+0xc1/0x1e0
Nov  8 04:32:00 whale12 kernel: [<ffffffff810ac65a>] ? tick_program_event+0x2a/0x30
Nov  8 04:32:00 whale12 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Nov  8 04:32:00 whale12 kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Nov  8 04:32:00 whale12 kernel: [<ffffffff8107a4a5>] ? irq_exit+0x85/0x90
Nov  8 04:32:00 whale12 kernel: [<ffffffff815325ea>] ? smp_apic_timer_interrupt+0x4a/0x60
Nov  8 04:32:00 whale12 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Nov  8 04:32:00 whale12 kernel: <EOI>  [<ffffffff81315e64>] ? acpi_idle_enter_simple+0x114/0x14b
Nov  8 04:32:00 whale12 kernel: [<ffffffff81315e60>] ? acpi_idle_enter_simple+0x110/0x14b
Nov  8 04:32:00 whale12 kernel: [<ffffffff81427567>] ? cpuidle_idle_call+0xa7/0x140
Nov  8 04:32:00 whale12 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Nov  8 04:32:00 whale12 kernel: [<ffffffff8152226c>] ? start_secondary+0x2ac/0x2ef
Nov  8 04:32:00 whale12 kernel: ---[ end trace 38cc645892010c1a ]---
Nov  8 04:32:00 whale12 kernel: tg3 0000:02:00.0: eth2: transmit timed out, resetting
Nov  8 04:32:00 whale12 kernel: hrtimer: interrupt took 3294565 ns
Nov  8 04:32:00 whale12 kernel: CE: hpet increasing min_delta_ns to 135758 nsec
Nov  8 04:32:00 whale12 kernel: CE: hpet increasing min_delta_ns to 203636 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 305454 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 458180 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 687270 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 1030904 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 1546356 nsec
Nov  8 04:32:01 whale12 kernel: CE: hpet increasing min_delta_ns to 2319534 nsec
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000000: 0x164314e4, 0x00100406, 0x02000010, 0x00800010
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000010: 0xd006000c, 0x00000000, 0xd005000c, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000020: 0xd004000c, 0x00000000, 0x00000000, 0x89a2152d
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000030: 0xff500000, 0x00000048, 0x00000000, 0x0000010b
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000040: 0x00000000, 0x1f000000, 0xc8035001, 0x16002008
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000060: 0x00000000, 0x00000000, 0xf1000298, 0x761b000f
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000070: 0x00071030, 0xd6000700, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000080: 0x164314e4, 0x800000f8, 0x00000000, 0x00000ffe
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000090: 0x00000000, 0x00000002, 0x00000000, 0x000003e7
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000000a0: 0x8005ac11, 0x00000004, 0x00000122, 0x00020010
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000000b0: 0x112c8d82, 0x00105400, 0x00475c12, 0x10120043
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000000d0: 0x0008081f, 0x00000000, 0x00000000, 0x00010002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000000f0: 0x00000000, 0x05762100, 0x00000000, 0x47708810
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000110: 0x00002000, 0x00002000, 0x000000a0, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000140: 0x01dbdaae, 0x0000089e, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000001
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000160: 0x1b010002, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000170: 0x00000000, 0x80000001, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000001b0: 0x23010018, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000200: 0x00000000, 0x1f000000, 0x00000000, 0x32000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000210: 0x00000000, 0x49000000, 0x00000000, 0x8f000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000220: 0x00000000, 0x6a000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000003e7
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000280: 0x00000000, 0x00000ffe, 0x00000000, 0x00000df1
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000290: 0x00000000, 0x00000427, 0x00000000, 0x00000109
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000300: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000310: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000320: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000330: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000340: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000350: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000360: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000370: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000380: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000390: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003a0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003b0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003c0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003d0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003e0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000003f0: 0x00000000, 0x00000002, 0x00000000, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000400: 0x19e04808, 0x00000000, 0x00001000, 0xfff85c80
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000410: 0x0000089e, 0x01dbdaae, 0x0000089e, 0x01dbdaae
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000420: 0x0000089e, 0x01dbdaae, 0x0000089e, 0x01dbdaae
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000430: 0x00000000, 0x00000000, 0x00000291, 0x000005f2
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x0825c1e1
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000460: 0x00000008, 0x00002620, 0x03ff0002, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000470: 0x80000000, 0x00000000, 0x00080000, 0x40000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000540: 0x0000089e, 0x01dbdaaf, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000590: 0x00900110, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x0000000c
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000610: 0xffffffff, 0x00000000, 0x00000004, 0x30000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000670: 0x5f865437, 0xe4ac62cc, 0x50103a45, 0x36621985
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000680: 0xbf14c0e8, 0x1bc27a1e, 0x84f4b556, 0x094ea6fe
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000690: 0x7dda01e7, 0xc04d7481, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000880: 0x00000d34, 0x03811964, 0x00000000, 0x00000014
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000890: 0x00000001, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000900: 0x397f06a0, 0xffffffff, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x9c2cf543
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000970: 0x000052f7, 0x00019ad7, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000980: 0x639d31d0, 0x03811964, 0x00000000, 0x88104959
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000990: 0x001a36c3, 0x00279d76, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x002a0000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000c80: 0x76ddc2b4, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000ce0: 0x78473602, 0x0000000c, 0x0000002a, 0x00040028
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00000cf0: 0x00000000, 0x50000002, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001000: 0x00000002, 0x00000000, 0xa0007f7f, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001010: 0x00020021, 0x00007f7f, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000001
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001440: 0x00000002, 0x00000002, 0x00000002, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001450: 0x00000002, 0x00000002, 0x00000002, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001460: 0x00000002, 0x00000002, 0x00000002, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001470: 0x00000002, 0x00000002, 0x00000002, 0x00000002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001800: 0x00000016, 0x00000000, 0x00000002, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001850: 0x0000001f, 0x00000004, 0x00004010, 0x00020002
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001860: 0x01000000, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002000: 0x00000002, 0x00000010, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002010: 0x00000181, 0x00000001, 0x007bfffd, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002100: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002110: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002120: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002130: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002140: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002150: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002160: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002170: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002180: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002190: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021a0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021b0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021c0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021d0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021e0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000021f0: 0x000c3a25, 0x000d9edb, 0x0000000d, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002200: 0x8856eaf6, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002250: 0x001a4a69, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002400: 0x00010012, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002450: 0x00000004, 0x77350000, 0x08001800, 0x00006000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002470: 0x00000000, 0x00000330, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000024c0: 0x0201a101, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x000003a5
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002c20: 0x00020001, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x000003a5
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003600: 0x00034000, 0x00130000, 0x00110000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003610: 0x00131100, 0x00000000, 0x00130000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003620: 0x80110011, 0x00000000, 0x00000000, 0x3c822080
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003630: 0x00000000, 0x80008000, 0x80001080, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003640: 0x0000000a, 0x00400000, 0x00000020, 0x00000019
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003650: 0x00000171, 0x00000bff, 0x05762100, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003660: 0x00000000, 0x00000000, 0x000400a3, 0x00000202
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003670: 0x0000002a, 0xfeff7fe3, 0x0000000a, 0x00000080
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036a0: 0x00000000, 0x00000000, 0x00000000, 0x00000400
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036b0: 0x00000000, 0x07ff003f, 0x07ff07ff, 0x01000004
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036c0: 0xffffffff, 0x00000000, 0x00000000, 0x5f0e9297
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036e0: 0x80000b19, 0x80800899, 0x00500899, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000036f0: 0x19090900, 0x13180600, 0x00000301, 0x00000001
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003700: 0x00000000, 0x00000000, 0x00000000, 0x00001388
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003710: 0x87fffffd, 0x00000000, 0x00000000, 0x000010dc
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003720: 0x00000000, 0x00002040, 0x88006000, 0xa0002000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003750: 0x00000000, 0x00000000, 0x0000080b, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003780: 0x0000b7f9, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c00: 0x00000306, 0x00000000, 0x00000000, 0x00000048
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c30: 0x00000000, 0x00000000, 0x00000004, 0x78f68000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c50: 0x00000000, 0x00000325, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003c80: 0x00000001, 0x00000df1, 0x0000042a, 0x00000109
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003cc0: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003d00: 0x00000004, 0x7766a000, 0x00000004, 0x79598000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003d10: 0x00000004, 0x774db000, 0x00000004, 0x78fa2000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:02 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004010: 0x00000000, 0x002db012, 0x002ee610, 0x01000422
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004030: 0x00000000, 0x01080220, 0x002ee010, 0x0046d002
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004040: 0x00479000, 0x00000000, 0x00000010, 0x0047f062
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004400: 0x00000016, 0x00000000, 0x00010000, 0x0000a000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004420: 0x000000fd, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x0f000881
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004450: 0x025e0411, 0x00ee00dd, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004700: 0x00030002, 0x00100000, 0x00100010, 0x90650000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004710: 0x795c1ff0, 0x00065494, 0x00100012, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004720: 0x00000000, 0x00000010, 0xf02c0000, 0x795c0020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004770: 0x000c0404, 0x00000002, 0x00000010, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004800: 0x380303fe, 0x00000000, 0x00000000, 0x00000120
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004810: 0x00000000, 0x0000000a, 0x04069f78, 0x00002a28
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004820: 0x00000079, 0x00000000, 0xf0330000, 0x771af750
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004840: 0x444b6a56, 0x444b6a56, 0x000e2200, 0x79423cda
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004850: 0x80ad761f, 0x877705ea, 0x6d6c6d6c, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004860: 0x00000079, 0x11000000, 0x001ff800, 0x56000008
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004870: 0x05ea0081, 0x003e1820, 0x003e1820, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004890: 0x280c0c04, 0x00305400, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000048a0: 0x000f0010, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004900: 0x18030002, 0x00000003, 0x30000000, 0x00000120
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004910: 0x00000040, 0x00000003, 0x0000df7c, 0x00000018
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004920: 0x00000080, 0x023f8000, 0x78473001, 0x0f000030
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004930: 0x78473201, 0x0f000030, 0x78473601, 0x0f000030
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004940: 0x785b3e01, 0x0f000030, 0xa9a95151, 0x17177878
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004950: 0xf0330000, 0x78473030, 0xaf000000, 0x807f007f
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004960: 0x00000000, 0x78473230, 0xaf000000, 0x40000002
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004970: 0x00028202, 0x00205400, 0x0000001c, 0x000000ff
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004980: 0x00000080, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004990: 0x00000000, 0x00000000, 0x78473630, 0xaf000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000049a0: 0x00f000db, 0x00000080, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000049b0: 0x77860e01, 0x4a49ec01, 0x78473401, 0x7a7c7e01
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000049c0: 0x78473002, 0x78473202, 0x78473602, 0x785b3e02
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000049d0: 0x77860e02, 0x4a49ec02, 0x78473402, 0x7a7c7e02
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000049f0: 0xf0330000, 0x785b3e30, 0xaf000000, 0x0000ffff
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c00: 0x200003fe, 0x00000020, 0x00000000, 0x0000000a
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c10: 0x0000001a, 0x00000001, 0x00000003, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c20: 0x00000000, 0x00000046, 0x00000046, 0x00000006
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c30: 0x00000000, 0x003ff800, 0x000000b3, 0x000000b3
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c40: 0x80090000, 0x0df10000, 0x01090427, 0x0df00000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c50: 0x0108042a, 0x031f0fff, 0x80000000, 0x08261f1f
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00004c60: 0x00000046, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08000100
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005020: 0x3c04ff00, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005030: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005040: 0x00000000, 0x00000000, 0x08000f38, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08000f38
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000050a0: 0x30420080, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000050b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000050c0: 0x00000000, 0x00000000, 0x08000300, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08000c48
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005120: 0x30630001, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005130: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005140: 0x00000000, 0x00000000, 0x08000d94, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000fac
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000051a0: 0x30840fff, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000051b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000051c0: 0x00000000, 0x00000000, 0x08000d94, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005200: 0x14830018, 0x00000b50, 0x20000000, 0x5060001d
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005210: 0x00000000, 0x00000000, 0x03608821, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005220: 0x00008010, 0x14830018, 0x21120010, 0x08000f3c
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005230: 0x08000000, 0x08007f80, 0x00000000, 0x3c038000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005240: 0x00000000, 0x03608821, 0x14400016, 0xc0010000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005250: 0x06000000, 0x00000000, 0x00008010, 0x08000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005260: 0x00000000, 0x00000000, 0x03608821, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005270: 0x3c038000, 0xc0000000, 0x08007f80, 0x08007f48
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005280: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x0800010c
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000052a0: 0x8f643658, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000052b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000052c0: 0x00000000, 0x00000000, 0x08001bb2, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08001bdc
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005320: 0x3c020018, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005330: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005340: 0x00000000, 0x00000000, 0x08001408, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x080002f8
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000053a0: 0x00621024, 0x00000000, 0x00000000, 0x40000020
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000053b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000053c0: 0x00000000, 0x00000000, 0x0800115e, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005800: 0x00000000, 0x1f000000, 0x00000000, 0x32000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005810: 0x00000000, 0x49000000, 0x00000000, 0x8f000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005820: 0x00000000, 0x6a000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x000003e7
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005880: 0x00000000, 0x00000ffe, 0x00000000, 0x00000df1
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005890: 0x00000000, 0x00000427, 0x00000000, 0x00000109
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005900: 0x00000000, 0x00000002, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005980: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006000: 0x20010082, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006400: 0x00000000, 0x00000000, 0x00010791, 0xc0000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006410: 0x05000016, 0x05000016, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006430: 0x00000000, 0x14e41643, 0x89a2152d, 0x10020000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000040
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000064c0: 0x00000005, 0x00000004, 0x00000122, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000064d0: 0x00000040, 0x10008d82, 0x00000000, 0x00d75e12
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000064e0: 0x00000031, 0x0008001f, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006500: 0x03e10003, 0x01dbdaae, 0x0000089e, 0x00000003
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006530: 0x00000001, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006540: 0x0028081f, 0x0001001e, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006550: 0x00000001, 0x02800000, 0x0000000f, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006560: 0x0000000f, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000065f0: 0x00000000, 0x00000059, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006800: 0x14130034, 0x20081082, 0x000fff18, 0x47392bd1
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006810: 0x21120010, 0xffffffff, 0x00000000, 0x00000040
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006880: 0x77fff020, 0x00000040, 0x00801643, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00006890: 0x00800000, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000068c0: 0x00000044, 0x00000000, 0x00000000, 0x00000000
Nov  8 04:32:03 whale12 kernel: tg3 0000:02:00.0: eth2: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x000033cc
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 0x00007510: 0x00000000, 0xefdff7df, 0x00000000, 0x00000000
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 0: Host status block [00000000:00000000:(0000:0000:0000):(0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 0: NAPI info [00000000:00000000:(01ec:0000:01ff):0000:(00c8:0000:0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 1: Host status block [00000000:00000000:(0000:0000:0000):(0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 1: NAPI info [00000000:00000000:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 2: Host status block [00000000:00000000:(0000:0000:0000):(0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 2: NAPI info [00000000:00000000:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 3: Host status block [00000000:00000000:(0000:0000:0000):(0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 3: NAPI info [00000000:00000000:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 4: Host status block [00000000:00000000:(0000:0000:0000):(0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: 4: NAPI info [00000000:00000000:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: tg3_stop_block timed out, ofs=4c00 enable_bit=2
Nov  9 03:07:04 whale12 kernel: tg3 0000:02:00.0: eth2: Link is down
Nov  9 03:07:07 whale12 kernel: tg3 0000:02:00.0: eth2: Link is up at 1000 Mbps, full duplex
Nov  9 03:07:07 whale12 kernel: tg3 0000:02:00.0: eth2: Flow control is off for TX and off for RX
Nov  9 03:07:07 whale12 kernel: tg3 0000:02:00.0: eth2: EEE is disabled

Comment 24 Janus Troelsen 2015-09-08 16:54:33 UTC

I have a similar issue. I do not know if it is the same, but some pecularities of my errors messages are:

[28499.505795] tg3 0000:09:00.0 eth0: 0: NAPI info [00000084:00000084:(01be:01ba:01ff):0035:(00fd:0000:0000:0000)]
[28499.609888] tg3 0000:09:00.0: tg3_stop_block timed out, ofs=1400 enable_bit=2
[28499.713455] tg3 0000:09:00.0: tg3_stop_block timed out, ofs=c00 enable_bit=2
[28499.734396] tg3 0000:09:00.0 eth0: Link is down
[28502.109842] tg3 0000:09:00.0 eth0: Link is up at 100 Mbps, full duplex
[28502.109859] tg3 0000:09:00.0 eth0: Flow control is on for TX and on for RX
[28507.776511] tg3 0000:09:00.0 eth0: transmit timed out, resetting
[28509.489027] CE: Reprogramming failure. Giving up

(I do not see reprogramming failures in this thread)

The backtrace:

Sep  8 18:14:12 Fugl kernel: [27407.627909] ------------[ cut here ]------------
Sep  8 18:14:12 Fugl kernel: [27407.627929] WARNING: CPU: 4 PID: 0 at /home/kernel/COD/linux/net/sched/sch_generic.c:303 dev_watchdog+0x203/0x210()
Sep  8 18:14:12 Fugl kernel: [27407.627936] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Sep  8 18:14:12 Fugl kernel: [27407.627939] Modules linked in: ppp_mppe ppp_async crc_ccitt pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) rfc
omm bnep dm_crypt arc4 iwldvm uvcvideo mac80211 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media btusb btrtl btbcm btintel bluetooth iwlwifi snd_hda_codec_hdmi acer_wmi snd_hda_codec_realtek sparse_keymap snd_hda_codec_generic nfsd snd_hda_intel snd_seq_midi snd_hda_codec cfg80211 snd_hda_core auth_rpcgss coretemp snd_seq_midi_event snd_rawmidi snd_hwdep nfs_acl snd_seq i7core_edac nfs kvm_intel snd_pcm snd_seq_device kvm snd_timer shpchp pcspkr joydev snd lpc_ich soundcore lockd serio_raw grace intel_ips edac_core sunrpc mac_hid fscache binfmt_misc parport_pc ppdev lp parport btrfs xor raid6_pq nouveau i2c_algo_bit tg3 ttm drm_kms_helper hid_generic ahci ptp libahci usbhid mxm_wmi psmouse hid drm pps_core wmi video
Sep  8 18:14:12 Fugl kernel: [27407.628090] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G           OE   4.2.0-040200rc1-generic #201507051635
Sep  8 18:14:12 Fugl kernel: [27407.628096] Hardware name: Acer             Aspire 8940G     /SM80-CP         , BIOS V1.3117 04/25/2011
Sep  8 18:14:12 Fugl kernel: [27407.628101]  00000000 00000000 ef779ec0 c170b1b2 ef779f00 ef779ef0 c1064a87 c19d4244
Sep  8 18:14:12 Fugl kernel: [27407.628114]  ef779f1c 00000000 c19d4280 0000012f c163b193 c163b193 00000000 f5d70000
Sep  8 18:14:12 Fugl kernel: [27407.628127]  fffffd29 ef779f08 c1064af3 00000009 ef779f00 c19d4244 ef779f1c ef779f44
Sep  8 18:14:12 Fugl kernel: [27407.628140] Call Trace:
Sep  8 18:14:12 Fugl kernel: [27407.628154]  [<c170b1b2>] dump_stack+0x41/0x52
Sep  8 18:14:12 Fugl kernel: [27407.628163]  [<c1064a87>] warn_slowpath_common+0x87/0xc0
Sep  8 18:14:12 Fugl kernel: [27407.628172]  [<c163b193>] ? dev_watchdog+0x203/0x210
Sep  8 18:14:12 Fugl kernel: [27407.628179]  [<c163b193>] ? dev_watchdog+0x203/0x210
Sep  8 18:14:12 Fugl kernel: [27407.628186]  [<c1064af3>] warn_slowpath_fmt+0x33/0x40
Sep  8 18:14:12 Fugl kernel: [27407.628194]  [<c163b193>] dev_watchdog+0x203/0x210
Sep  8 18:14:12 Fugl kernel: [27407.628206]  [<c10c3510>] call_timer_fn+0x30/0xf0
Sep  8 18:14:12 Fugl kernel: [27407.628217]  [<c1099340>] ? rebalance_domains+0x150/0x260
Sep  8 18:14:12 Fugl kernel: [27407.628226]  [<c163af90>] ? dev_deactivate_queue.constprop.39+0x50/0x50
Sep  8 18:14:12 Fugl kernel: [27407.628234]  [<c10c4a29>] run_timer_softirq+0x189/0x230
Sep  8 18:14:12 Fugl kernel: [27407.628243]  [<c163af90>] ? dev_deactivate_queue.constprop.39+0x50/0x50
Sep  8 18:14:12 Fugl kernel: [27407.628252]  [<c1068723>] __do_softirq+0xb3/0x220
Sep  8 18:14:12 Fugl kernel: [27407.628261]  [<c1068670>] ? cpu_callback+0x160/0x160
Sep  8 18:14:12 Fugl kernel: [27407.628273]  [<c10132e2>] do_softirq_own_stack+0x22/0x30
Sep  8 18:14:12 Fugl kernel: [27407.628277]  <IRQ>  [<c1068ab5>] irq_exit+0x95/0xa0
Sep  8 18:14:12 Fugl kernel: [27407.628289]  [<c1712bf5>] do_IRQ+0x45/0xd0
Sep  8 18:14:12 Fugl kernel: [27407.628299]  [<c1712073>] common_interrupt+0x33/0x38
Sep  8 18:14:12 Fugl kernel: [27407.628310]  [<c15c82d1>] ? cpuidle_enter_state+0xb1/0x290
Sep  8 18:14:12 Fugl kernel: [27407.628319]  [<c15c84e4>] cpuidle_enter+0x14/0x20
Sep  8 18:14:12 Fugl kernel: [27407.628327]  [<c109ec13>] call_cpuidle+0x33/0x70
Sep  8 18:14:12 Fugl kernel: [27407.628335]  [<c109ee86>] cpu_startup_entry+0x236/0x2f0
Sep  8 18:14:12 Fugl kernel: [27407.628347]  [<c1044ad9>] start_secondary+0x169/0x1c0
Sep  8 18:14:12 Fugl kernel: [27407.628355] ---[ end trace 8739ccc4b339bf30 ]---
Sep  8 18:14:12 Fugl kernel: [27407.628366] tg3 0000:09:00.0 eth0: transmit timed out, resetting
Sep  8 18:14:13 Fugl kernel: [27408.124476] hrtimer: interrupt took 4367866 ns

Comment 25 anamm 2016-07-16 01:58:18 UTC

I have this issue in my SO Centos 7 kernel 3.10

[2041786.303995] ------------[ cut here ]------------
[2041786.304013] WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
[2041786.304016] NETDEV WATCHDOG: eno1 (tg3): transmit queue 0 timed out
[2041786.304018] Modules linked in: fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 binfmt_misc ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_
conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table
_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security
 iptable_raw iptable_filter intel_powerclamp coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ipmi_ssi
f cryptd sb_edac iTCO_wdt sr_mod cdrom edac_core hpwdt iTCO_vendor_support wmi usb_storage hpilo ipmi_si pcspkr ipmi_msghandler acpi_power_meter sg lpc_ich mf
d_core shpchp ioatdma
[2041786.304095]  dca pcc_cpufreq ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt i2c_algo
_bit drm_kms_helper ttm drm i2c_core crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix libata tg3 ptp hpsa pps_core dm_mirror dm_region_hash d
m_log dm_mod
[2041786.304129] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-327.18.2.el7.x86_64 #1
[2041786.304132] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015
[2041786.304135]  ffff880627a03d88 11d300e54b2f2116 ffff880627a03d40 ffffffff81635a0c
[2041786.304143]  ffff880627a03d78 ffffffff8107b200 0000000000000000 ffff88061d6ac000
[2041786.304149]  ffff88061d7c7580 0000000000000005 0000000000000000 ffff880627a03de0
[2041786.304155] Call Trace:
[2041786.304158]  <IRQ>  [<ffffffff81635a0c>] dump_stack+0x19/0x1b
[2041786.304175]  [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[2041786.304179]  [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[2041786.304189]  [<ffffffff8154d030>] dev_watchdog+0x270/0x280
[2041786.304194]  [<ffffffff8154cdc0>] ? dev_graft_qdisc+0x80/0x80
[2041786.304202]  [<ffffffff8108b0a6>] call_timer_fn+0x36/0x110
[2041786.304207]  [<ffffffff8154cdc0>] ? dev_graft_qdisc+0x80/0x80
[2041786.304213]  [<ffffffff8108dd97>] run_timer_softirq+0x237/0x340
[2041786.304218]  [<ffffffff81084b0f>] __do_softirq+0xef/0x280
[2041786.304223]  [<ffffffff81647adc>] call_softirq+0x1c/0x30
[2041786.304233]  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[2041786.304238]  [<ffffffff81084ea5>] irq_exit+0x115/0x120
[2041786.304244]  [<ffffffff81648755>] smp_apic_timer_interrupt+0x45/0x60
[2041786.304251]  [<ffffffff81646e1d>] apic_timer_interrupt+0x6d/0x80
[2041786.304253]  <EOI>  [<ffffffff814d49e2>] ? cpuidle_enter_state+0x52/0xc0
[2041786.304267]  [<ffffffff814d49d8>] ? cpuidle_enter_state+0x48/0xc0
[2041786.304273]  [<ffffffff814d4b29>] cpuidle_idle_call+0xd9/0x210
[2041786.304280]  [<ffffffff8101e4be>] arch_cpu_idle+0xe/0x30
[2041786.304287]  [<ffffffff810d6445>] cpu_startup_entry+0x245/0x290
[2041786.304295]  [<ffffffff81625677>] rest_init+0x77/0x80
[2041786.304304]  [<ffffffff81a8d057>] start_kernel+0x429/0x44a
[2041786.304310]  [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
[2041786.304316]  [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
[2041786.304322]  [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
[2041786.304327]  [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
[2041786.304330] ---[ end trace 5fafdeb818cf662c ]---
[2041786.304336] tg3 0000:03:00.0 eno1: transmit timed out, resetting
[2041787.563458] tg3 0000:03:00.0 eno1: 0x00000000: 0x165714e4, 0x00100446, 0x02000001, 0x00800010
[2041787.563481] tg3 0000:03:00.0 eno1: 0x00000010: 0xf6bf000c, 0x00000000, 0xf6be000c, 0x00000000
[2041787.563503] tg3 0000:03:00.0 eno1: 0x00000020: 0xf6bd000c, 0x00000000, 0x00000000, 0x169d103c
[2041787.563523] tg3 0000:03:00.0 eno1: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000105
[2041787.563544] tg3 0000:03:00.0 eno1: 0x00000040: 0x00000000, 0x6e000000, 0xc8035001, 0x64002008
[2041787.563567] tg3 0000:03:00.0 eno1: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[2041787.563586] tg3 0000:03:00.0 eno1: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x00380081
[2041787.563608] tg3 0000:03:00.0 eno1: 0x00000070: 0x000710b0, 0xb0400600, 0x00000000, 0x00000000
[2041787.563632] tg3 0000:03:00.0 eno1: 0x00000080: 0x165714e4, 0xc00000d0, 0x00000000, 0x0000071f
[2041787.563653] tg3 0000:03:00.0 eno1: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000000ab
[2041787.564021] tg3 0000:03:00.0 eno1: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
[2041787.564388] tg3 0000:03:00.0 eno1: 0x000000b0: 0x10008d81, 0x00105426, 0x0004cc42, 0x10420040
[2041787.564756] tg3 0000:03:00.0 eno1: 0x000000d0: 0x0000001f, 0x00000000, 0x00000000, 0x00000001
[2041787.565121] tg3 0000:03:00.0 eno1: 0x000000f0: 0x00000000, 0x05719001, 0x00000000, 0x96002600
[2041787.565488] tg3 0000:03:00.0 eno1: 0x00000100: 0x13c10001, 0x00100000, 0x00100000, 0x00000020
[2041787.565854] tg3 0000:03:00.0 eno1: 0x00000110: 0x00002000, 0x00000000, 0x000000a0, 0x04000001
[2041787.566215] tg3 0000:03:00.0 eno1: 0x00000120: 0x0010000f, 0x03070000, 0x00000000, 0x00000000
[2041787.566580] tg3 0000:03:00.0 eno1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[2041787.566944] tg3 0000:03:00.0 eno1: 0x00000140: 0xf020c180, 0x000040a8, 0x00000000, 0x00000000
[2041787.567307] tg3 0000:03:00.0 eno1: 0x00000150: 0x16010004, 0x00000000, 0x0007812c, 0x00000001
[2041787.567677] tg3 0000:03:00.0 eno1: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
[2041787.568040] tg3 0000:03:00.0 eno1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[2041787.568406] tg3 0000:03:00.0 eno1: 0x00000200: 0x00000000, 0x6e000000, 0x00000000, 0x29000000
[2041787.568783] tg3 0000:03:00.0 eno1: 0x00000210: 0x00000000, 0xeb000000, 0x00000000, 0x03000000
[2041787.569148] tg3 0000:03:00.0 eno1: 0x00000220: 0x00000000, 0xd4000000, 0x00000000, 0x00000000
[2041787.569516] tg3 0000:03:00.0 eno1: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000000ab
[2041787.569883] tg3 0000:03:00.0 eno1: 0x00000280: 0x00000000, 0x0000071f, 0x00000000, 0x00000eca
[2041787.570246] tg3 0000:03:00.0 eno1: 0x00000290: 0x00000000, 0x00000ba0, 0x00000000, 0x00000e5a
[2041787.570612] tg3 0000:03:00.0 eno1: 0x00000300: 0x00000000, 0x00000122, 0x00000000, 0x00000000
[2041787.570979] tg3 0000:03:00.0 eno1: 0x00000400: 0x18e04806, 0x02400000, 0x00001000, 0x00000900
[2041787.571345] tg3 0000:03:00.0 eno1: 0x00000410: 0x000040a8, 0xf020c180, 0x000040a8, 0xf020c180
[2041787.571713] tg3 0000:03:00.0 eno1: 0x00000420: 0x000040a8, 0xf020c180, 0x000040a8, 0xf020c180
[2041787.572078] tg3 0000:03:00.0 eno1: 0x00000430: 0x00000400, 0x00000000, 0x000001f7, 0x000005f2
[2041787.572444] tg3 0000:03:00.0 eno1: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x082a0000
[2041787.572811] tg3 0000:03:00.0 eno1: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
:
[2041787.573187] tg3 0000:03:00.0 eno1: 0x00000460: 0x00000018, 0x00002620, 0x01ff0002, 0x00000000
[2041787.573550] tg3 0000:03:00.0 eno1: 0x00000470: 0x80000000, 0x00800000, 0x00000000, 0x40000000
[2041787.573919] tg3 0000:03:00.0 eno1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[2041787.574285] tg3 0000:03:00.0 eno1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[2041787.574653] tg3 0000:03:00.0 eno1: 0x00000540: 0x0000c434, 0x6bbc0e91, 0x00000000, 0x00000000
[2041787.575019] tg3 0000:03:00.0 eno1: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[2041787.575385] tg3 0000:03:00.0 eno1: 0x000005b0: 0x00000000, 0xc0000000, 0x00000000, 0x00000000
[2041787.575753] tg3 0000:03:00.0 eno1: 0x000005c0: 0xe247f120, 0xdd20104a, 0x00000000, 0x00000000
[2041787.576118] tg3 0000:03:00.0 eno1: 0x00000600: 0xffffffff, 0x00f800d1, 0x00000000, 0x00001f04
[2041787.576482] tg3 0000:03:00.0 eno1: 0x00000610: 0xffffffff, 0x00000000, 0x07c00244, 0x00000000
[2041787.576851] tg3 0000:03:00.0 eno1: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[2041787.577213] tg3 0000:03:00.0 eno1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.577581] tg3 0000:03:00.0 eno1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.577949] tg3 0000:03:00.0 eno1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.578313] tg3 0000:03:00.0 eno1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.578683] tg3 0000:03:00.0 eno1: 0x00000670: 0xfeb8bed8, 0x47d7204f, 0x6a3e4143, 0x7ffb4c55
[2041787.579045] tg3 0000:03:00.0 eno1: 0x00000680: 0x383e53bc, 0x9ccc584d, 0xc0c6d64a, 0x38c28137
[2041787.579411] tg3 0000:03:00.0 eno1: 0x00000690: 0x543d36ea, 0x490674f4, 0x00000000, 0x00000000
[2041787.579779] tg3 0000:03:00.0 eno1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[2041787.580143] tg3 0000:03:00.0 eno1: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.580506] tg3 0000:03:00.0 eno1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.580873] tg3 0000:03:00.0 eno1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[2041787.581234] tg3 0000:03:00.0 eno1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.581599] tg3 0000:03:00.0 eno1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.581969] tg3 0000:03:00.0 eno1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.582332] tg3 0000:03:00.0 eno1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[2041787.582700] tg3 0000:03:00.0 eno1: 0x00000880: 0x000001e2, 0x0007b78a, 0x00000000, 0x00000005
[2041787.583066] tg3 0000:03:00.0 eno1: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[2041787.583432] tg3 0000:03:00.0 eno1: 0x00000900: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.583798] tg3 0000:03:00.0 eno1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.584164] tg3 0000:03:00.0 eno1: 0x00000920: 0x00000000, 0x000134f0, 0xffffffff, 0x00000000
[2041787.584528] tg3 0000:03:00.0 eno1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.584895] tg3 0000:03:00.0 eno1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.585261] tg3 0000:03:00.0 eno1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.585626] tg3 0000:03:00.0 eno1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[2041787.585994] tg3 0000:03:00.0 eno1: 0x00000980: 0x00001146, 0x0007b78a, 0x00000000, 0x0000002f
[2041787.586360] tg3 0000:03:00.0 eno1: 0x00000990: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[2041787.586725] tg3 0000:03:00.0 eno1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[2041787.587093] tg3 0000:03:00.0 eno1: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x00560000
[2041787.587460] tg3 0000:03:00.0 eno1: 0x00000c80: 0x0262c6c1, 0x00000000, 0x00000000, 0x00000000
:
[2041787.573187] tg3 0000:03:00.0 eno1: 0x00000460: 0x00000018, 0x00002620, 0x01ff0002, 0x00000000
[2041787.573550] tg3 0000:03:00.0 eno1: 0x00000470: 0x80000000, 0x00800000, 0x00000000, 0x40000000
[2041787.573919] tg3 0000:03:00.0 eno1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[2041787.574285] tg3 0000:03:00.0 eno1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[2041787.574653] tg3 0000:03:00.0 eno1: 0x00000540: 0x0000c434, 0x6bbc0e91, 0x00000000, 0x00000000
[2041787.575019] tg3 0000:03:00.0 eno1: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[2041787.575385] tg3 0000:03:00.0 eno1: 0x000005b0: 0x00000000, 0xc0000000, 0x00000000, 0x00000000
[2041787.575753] tg3 0000:03:00.0 eno1: 0x000005c0: 0xe247f120, 0xdd20104a, 0x00000000, 0x00000000
[2041787.576118] tg3 0000:03:00.0 eno1: 0x00000600: 0xffffffff, 0x00f800d1, 0x00000000, 0x00001f04
[2041787.576482] tg3 0000:03:00.0 eno1: 0x00000610: 0xffffffff, 0x00000000, 0x07c00244, 0x00000000
[2041787.576851] tg3 0000:03:00.0 eno1: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[2041787.577213] tg3 0000:03:00.0 eno1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.577581] tg3 0000:03:00.0 eno1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.577949] tg3 0000:03:00.0 eno1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.578313] tg3 0000:03:00.0 eno1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[2041787.578683] tg3 0000:03:00.0 eno1: 0x00000670: 0xfeb8bed8, 0x47d7204f, 0x6a3e4143, 0x7ffb4c55
[2041787.579045] tg3 0000:03:00.0 eno1: 0x00000680: 0x383e53bc, 0x9ccc584d, 0xc0c6d64a, 0x38c28137
[2041787.579411] tg3 0000:03:00.0 eno1: 0x00000690: 0x543d36ea, 0x490674f4, 0x00000000, 0x00000000
[2041787.579779] tg3 0000:03:00.0 eno1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[2041787.580143] tg3 0000:03:00.0 eno1: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.580506] tg3 0000:03:00.0 eno1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.580873] tg3 0000:03:00.0 eno1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[2041787.581234] tg3 0000:03:00.0 eno1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.581599] tg3 0000:03:00.0 eno1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.581969] tg3 0000:03:00.0 eno1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.582332] tg3 0000:03:00.0 eno1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[2041787.582700] tg3 0000:03:00.0 eno1: 0x00000880: 0x000001e2, 0x0007b78a, 0x00000000, 0x00000005
[2041787.583066] tg3 0000:03:00.0 eno1: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[2041787.583432] tg3 0000:03:00.0 eno1: 0x00000900: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.583798] tg3 0000:03:00.0 eno1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[2041787.584164] tg3 0000:03:00.0 eno1: 0x00000920: 0x00000000, 0x000134f0, 0xffffffff, 0x00000000
[2041787.584528] tg3 0000:03:00.0 eno1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.584895] tg3 0000:03:00.0 eno1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.585261] tg3 0000:03:00.0 eno1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[2041787.585626] tg3 0000:03:00.0 eno1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[2041787.585994] tg3 0000:03:00.0 eno1: 0x00000980: 0x00001146, 0x0007b78a, 0x00000000, 0x0000002f
[2041787.586360] tg3 0000:03:00.0 eno1: 0x00000990: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[2041787.586725] tg3 0000:03:00.0 eno1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[2041787.587093] tg3 0000:03:00.0 eno1: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x00560000
[2041787.587460] tg3 0000:03:00.0 eno1: 0x00000c80: 0x0262c6c1, 0x00000000, 0x00000000, 0x00000000
:
[2041787.606603] tg3 0000:03:00.0 eno1: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[2041787.606969] tg3 0000:03:00.0 eno1: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[2041787.607334] tg3 0000:03:00.0 eno1: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[2041787.607702] tg3 0000:03:00.0 eno1: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[2041787.608065] tg3 0000:03:00.0 eno1: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[2041787.608425] tg3 0000:03:00.0 eno1: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[2041787.608788] tg3 0000:03:00.0 eno1: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[2041787.609151] tg3 0000:03:00.0 eno1: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[2041787.609514] tg3 0000:03:00.0 eno1: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[2041787.609881] tg3 0000:03:00.0 eno1: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[2041787.610247] tg3 0000:03:00.0 eno1: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[2041787.610611] tg3 0000:03:00.0 eno1: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[2041787.610980] tg3 0000:03:00.0 eno1: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[2041787.611346] tg3 0000:03:00.0 eno1: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
[2041787.611712] tg3 0000:03:00.0 eno1: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400
[2041787.612079] tg3 0000:03:00.0 eno1: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000
[2041787.612444] tg3 0000:03:00.0 eno1: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800
[2041787.612813] tg3 0000:03:00.0 eno1: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400
[2041787.613177] tg3 0000:03:00.0 eno1: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00
[2041787.613541] tg3 0000:03:00.0 eno1: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800
[2041787.613909] tg3 0000:03:00.0 eno1: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000
[2041787.614272] tg3 0000:03:00.0 eno1: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00
[2041787.614636] tg3 0000:03:00.0 eno1: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400
[2041787.615004] tg3 0000:03:00.0 eno1: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000
[2041787.615369] tg3 0000:03:00.0 eno1: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800
[2041787.615736] tg3 0000:03:00.0 eno1: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400
[2041787.616101] tg3 0000:03:00.0 eno1: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00
[2041787.616465] tg3 0000:03:00.0 eno1: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800
[2041787.616833] tg3 0000:03:00.0 eno1: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000
[2041787.617198] tg3 0000:03:00.0 eno1: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00
[2041787.617563] tg3 0000:03:00.0 eno1: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400
[2041787.617931] tg3 0000:03:00.0 eno1: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000
[2041787.618295] tg3 0000:03:00.0 eno1: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[2041787.618665] tg3 0000:03:00.0 eno1: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x00000063
[2041787.619031] tg3 0000:03:00.0 eno1: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[2041787.619396] tg3 0000:03:00.0 eno1: 0x00002c20: 0x00000004, 0x00000000, 0x00000000, 0x00000000
[2041787.619766] tg3 0000:03:00.0 eno1: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000
[2041787.620134] tg3 0000:03:00.0 eno1: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x00000063
[2041787.620499] tg3 0000:03:00.0 eno1: 0x00003400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[2041787.620866] tg3 0000:03:00.0 eno1: 0x00003600: 0x00004600, 0x00170000, 0x00110000, 0x00000000
:
[2041787.689148] tg3 0000:03:00.0 eno1: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[2041787.689514] tg3 0000:03:00.0 eno1: 0x00007030: 0x00020000, 0x00004c28, 0x001f0000, 0x00000000
[2041787.689887] tg3 0000:03:00.0 eno1: 0: Host status block [00000001:0000006e:(0000:076a:0000):(0000:0070)]
[2041787.690253] tg3 0000:03:00.0 eno1: 0: NAPI info [0000006e:0000006e:(0122:0070:01ff):0000:(00bf:0000:0000:0000)]
[2041787.690622] tg3 0000:03:00.0 eno1: 1: Host status block [00000001:00000029:(0000:0000:0000):(071f:0000)]
[2041787.690991] tg3 0000:03:00.0 eno1: 1: NAPI info [00000029:00000029:(0000:0000:01ff):071f:(071f:071f:0000:0000)]
[2041787.691359] tg3 0000:03:00.0 eno1: 2: Host status block [00000001:000000fd:(0edc:0000:0000):(0000:0000)]
[2041787.691732] tg3 0000:03:00.0 eno1: 2: NAPI info [000000fd:000000fd:(0000:0000:01ff):0edc:(06dc:06dc:0000:0000)]
[2041787.692101] tg3 0000:03:00.0 eno1: 3: Host status block [00000001:00000005:(0000:0000:0000):(0000:0000)]
[2041787.692469] tg3 0000:03:00.0 eno1: 3: NAPI info [00000003:00000003:(0000:0000:01ff):0ba0:(03a0:03a0:0000:0000)]
[2041787.692841] tg3 0000:03:00.0 eno1: 4: Host status block [00000001:000000d6:(0000:0000:0e5c):(0000:0000)]
[2041787.693208] tg3 0000:03:00.0 eno1: 4: NAPI info [000000d6:000000d6:(0000:0000:01ff):0e5c:(065c:065c:0000:0000)]
[2041787.803833] tg3 0000:03:00.0: tg3_stop_block timed out, ofs=1400 enable_bit=2
[2041787.906120] tg3 0000:03:00.0: tg3_stop_block timed out, ofs=c00 enable_bit=2
[2041787.983443] tg3 0000:03:00.0 eno1: Link is down
[2041790.066223] tg3 0000:03:00.0 eno1: Link is up at 100 Mbps, half duplex
[2041790.066235] tg3 0000:03:00.0 eno1: Flow control is off for TX and off for RX
[2041790.066240] tg3 0000:03:00.0 eno1: EEE is disabled

Comment 26 [account disabled by administrator] 2016-07-18 22:51:34 UTC

Anamm,
Can you try the latest mainline kernel as 3.10 is not currently supported by the kernel community therefore I would like to see if this is reproducible on a newer mainline kernel from the current, 4.7 release cycle.

Comment 27 Alexander Y. Tiurin 2016-11-18 21:24:41 UTC

I confirm this bug too. Kernel 4.8.7

[  220.127011] ------------[ cut here ]------------
[  220.127038] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0x21d/0x220
[  220.127041] NETDEV WATCHDOG: eth2 (tg3): transmit queue 0 timed out
[  220.127043] Modules linked in: xt_nat xt_recent xt_comment xt_AUDIT iptable_nat nf_nat_ipv4 xt_connmark xt_realm ipt_REJECT xt_addrtype xt_physdev br_netfilter bridge xt_mark xt_CT iptable_raw xt_multiport xt_LOG nf_log_ipv4 nf_log_common nf_nat_sip nf_nat_irc nf_nat_ftp nf_nat nf_conntrack_proto_sctp hpwdt sch_red 8021q garp mrp stp llc hpilo bnx2 cciss
[  220.127079] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.8.7-debug1 #6
[  220.127081] Hardware name: HP ProLiant DL380 G5, BIOS P56 05/01/2007
[  220.127083]  0000000000000000 ffff88007fcc3d90 ffffffff81314738 ffff88007fcc3de0
[  220.127086]  0000000000000000 ffff88007fcc3dd0 ffffffff8105b4a6 0000013c00000040
[  220.127089]  0000000000000000 ffff88007b32c000 0000000000000003 0000000000000005
[  220.127092] Call Trace:
[  220.127094]  <IRQ>  [<ffffffff81314738>] dump_stack+0x4d/0x65
[  220.127105]  [<ffffffff8105b4a6>] __warn+0xc6/0xe0
[  220.127107]  [<ffffffff8105b50a>] warn_slowpath_fmt+0x4a/0x50
[  220.127110]  [<ffffffff8173a3fd>] dev_watchdog+0x21d/0x220
[  220.127112]  [<ffffffff8173a1e0>] ? qdisc_rcu_free+0x40/0x40
[  220.127116]  [<ffffffff810bb370>] call_timer_fn+0x30/0x120
[  220.127119]  [<ffffffff810bb8b1>] run_timer_softirq+0x201/0x480
[  220.127122]  [<ffffffff810c31b7>] ? ktime_get+0x37/0xa0
[  220.127126]  [<ffffffff8103dd38>] ? lapic_next_event+0x18/0x20
[  220.127129]  [<ffffffff810c945d>] ? clockevents_program_event+0x7d/0x120
[  220.127133]  [<ffffffff810606bb>] __do_softirq+0xeb/0x270
[  220.127135]  [<ffffffff81060a28>] irq_exit+0x98/0xa0
[  220.127137]  [<ffffffff8103e8cd>] smp_apic_timer_interrupt+0x3d/0x50
[  220.127142]  [<ffffffff818a65ef>] apic_timer_interrupt+0x7f/0x90
[  220.127143]  <EOI>  [<ffffffff81025f1f>] ? mwait_idle+0x8f/0x160
[  220.127149]  [<ffffffff8102642a>] arch_cpu_idle+0xa/0x10
[  220.127152]  [<ffffffff81096135>] default_idle_call+0x25/0x30
[  220.127154]  [<ffffffff810963ba>] cpu_startup_entry+0x27a/0x2d0
[  220.127157]  [<ffffffff8103cbf8>] start_secondary+0xe8/0xf0
[  220.127162] ---[ end trace 2c8116174dc1811a ]---
[  220.127169] tg3 0000:06:00.1 eth2: transmit timed out, resetting
[  220.128003] tg3 0000:06:00.1 eth2: 0x00000000: 0x165f14e4, 0x00100446, 0x02000000, 0x00800010
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000010: 0xf7bc000c, 0x00000000, 0xf7bb000c, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000020: 0xf7ba000c, 0x00000000, 0x00000000, 0x1786103c
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x0000020a
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000040: 0x00000000, 0x30000000, 0xc8035001, 0x64002008
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000070: 0x000710b0, 0xffbaddff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000080: 0x165f14e4, 0x40000042, 0x00000000, 0x00000649
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x00000736
[  221.381002] tg3 0000:06:00.1 eth2: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
[  221.381002] tg3 0000:06:00.1 eth2: 0x000000b0: 0x10008d81, 0x00105426, 0x0004cc22, 0x10110040
[  221.381002] tg3 0000:06:00.1 eth2: 0x000000d0: 0x0000001f, 0x00000000, 0x00000000, 0x00010000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000100: 0x13c10001, 0x00100000, 0x00100000, 0x0017f030
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000110: 0x00000000, 0x000031c0, 0x000000a0, 0x05000001
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000120: 0x00181c03, 0x08f80000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000140: 0xfd8e8a9b, 0x00002c44, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000200: 0x00000000, 0x30000000, 0x00000000, 0xc1000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000210: 0x00000000, 0x3e000000, 0x00000000, 0x77000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000220: 0x00000000, 0x7f000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x00000736
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000280: 0x00000000, 0x00000649, 0x00000000, 0x000009fb
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000290: 0x00000000, 0x000003ad, 0x00000000, 0x00000a7d
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000300: 0x00000000, 0x00000184, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000400: 0x18e04806, 0x02400000, 0x00001000, 0x00000d00
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000410: 0x00002c44, 0xfd8e8a9b, 0x00002c44, 0xfd8e8a9b
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000420: 0x00002c44, 0xfd8e8a9b, 0x00002c44, 0xfd8e8a9b
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000430: 0x00000400, 0x00000000, 0x000000a9, 0x000005f2
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04584400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000460: 0x00000018, 0x00002620, 0x01ff0002, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000470: 0x00000000, 0x10000000, 0x00000000, 0x40000042
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000005c0: 0x5bbbbbe8, 0xe7d560b7, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000600: 0xffffffff, 0x00f80051, 0x00000000, 0x00001f04
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000610: 0xffffffff, 0x00000000, 0x07c00044, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000670: 0xe43ef12c, 0xa0f4458d, 0xd216bf6c, 0x4c9eb80d
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000680: 0xd3f7d46c, 0x64bc431e, 0x39ba2e80, 0xef0aac58
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000690: 0xf252af49, 0xc218b638, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000880: 0x0007ede0, 0x00090f21, 0x00000000, 0x000004a9
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000890: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000900: 0x012be2a4, 0xffffffff, 0x000024cb, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000910: 0x00000000, 0xffffffff, 0x0000007c, 0x00000d4a
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000920: 0x000080a3, 0x00000863, 0xffffffff, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000930: 0x00001ed7, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x00041291
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000970: 0x00000001, 0x00000003, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000980: 0x004df920, 0x00090f21, 0x00000000, 0x00049e65
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000990: 0x00000009, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x004c0000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000c80: 0x000432d9, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000ce0: 0x64a1f882, 0x00000000, 0x0000004c, 0x00040028
[  221.381002] tg3 0000:06:00.1 eth2: 0x00000cf0: 0x00000000, 0x50000034, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001000: 0x00000002, 0x00000000, 0xa000b3b3, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001010: 0x512d12e1, 0x0000b3b3, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001440: 0x00000135, 0x00000135, 0x00000135, 0x00000135
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001450: 0x00000135, 0x00000135, 0x00000135, 0x00000135
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001460: 0x00000135, 0x00000135, 0x00000135, 0x00000135
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001470: 0x00000135, 0x00000135, 0x00000135, 0x00000135
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001480: 0x00001114, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001800: 0x00000016, 0x00000000, 0x00000165, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0x75470000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001840: 0x75470000, 0x00000000, 0x10000211, 0xc0000001
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001850: 0x0000001f, 0x00000000, 0x00004150, 0x01650165
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001860: 0x10001900, 0x00000000, 0x75471550, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002100: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002110: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002120: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002130: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002140: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002150: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002160: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002170: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002180: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002190: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021a0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021b0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021c0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021d0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021e0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000021f0: 0x000decf6, 0x000decf6, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002200: 0x0004a4a0, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002400: 0x00010012, 0x00000000, 0x00246001, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002450: 0x00000000, 0x755b0000, 0x08001800, 0x00040000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002470: 0x00000000, 0x00000673, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400
[  221.381002] tg3 0000:06:00.1 eth2: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400
[  221.381002] tg3 0000:06:00.1 eth2: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x0000072a
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002c20: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x0000072a
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003440: 0x00000000, 0x00000000, 0x00000000, 0x00000110
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003600: 0x00000600, 0x00170000, 0x00110000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003610: 0x00170000, 0x00000000, 0x00130000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003620: 0x00110011, 0x00000000, 0x00000000, 0x400b2080
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003630: 0x00800000, 0x87748774, 0x02c01000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003640: 0x00000000, 0x00000000, 0x00000020, 0x00000019
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003650: 0x00000171, 0x000f03ff, 0x05720000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003660: 0x00000000, 0x00000000, 0x02000000, 0x00000202
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003670: 0x00000000, 0xfef7afe3, 0x00000000, 0x00000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003680: 0x30018010, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000036a0: 0x000001a0, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000036b0: 0x0010034c, 0x07ff07ff, 0x07ff07ff, 0x01000004
[  221.381002] tg3 0000:06:00.1 eth2: 0x000036c0: 0xfffffffe, 0x00000000, 0x00000000, 0x0004fb73
[  221.381002] tg3 0000:06:00.1 eth2: 0x000036d0: 0x000001f8, 0x00000000, 0x00000000, 0x00004d3b
[  221.381002] tg3 0000:06:00.1 eth2: 0x000036f0: 0x00000000, 0x00000000, 0x00000000, 0x00012901
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003800: 0x00000001, 0x00000000, 0x0000000e, 0x0516028b
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003810: 0x000001ff, 0x0000009f, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c00: 0x00000306, 0x00000000, 0x00000000, 0x00000048
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c30: 0x00000000, 0x00000000, 0x00000000, 0x7bc6b000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c40: 0x00000000, 0x00000b00, 0x00000040, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c50: 0x00000000, 0x0000072a, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003c80: 0x00000676, 0x00000a36, 0x000003d8, 0x00000aa6
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003cc0: 0x0000012d, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003d00: 0x00000000, 0x7c065000, 0x00000000, 0x7c207000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003d10: 0x00000000, 0x7c26e000, 0x00000000, 0x75582000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00003fc0: 0x000325d2, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004000: 0x00000002, 0x00000000, 0x0011c35a, 0x001f072f
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004010: 0x00000000, 0x002c0012, 0x00000490, 0x0084a842
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004030: 0x00000010, 0x0080a910, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004040: 0x00000000, 0x00000000, 0x01086620, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004050: 0x00000000, 0x00000000, 0x002e1010, 0x00418002
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004060: 0x00419000, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004400: 0x00000016, 0x00000000, 0x00010000, 0x0000a000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x091244b6
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004450: 0x02404c20, 0x00e200c2, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004800: 0x380303fe, 0x00000000, 0x00000000, 0x00000120
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004810: 0x00000000, 0x0000000b, 0x00ac97f8, 0x00002a28
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004820: 0x00000019, 0x00000000, 0xf0330000, 0x6127e3a0
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004830: 0x00000000, 0x00000283, 0x00000283, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004840: 0x000000b6, 0x000000b6, 0x000e2200, 0x1936815a
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004850: 0x0fd262ad, 0x736c005a, 0x18171817, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004860: 0x02000019, 0x113bcb72, 0x00100800, 0x42040008
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004870: 0x05ea4980, 0x003e1820, 0x003e1820, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004900: 0x28190404, 0x00305407, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x9eab0000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a10: 0x755be4e0, 0x008ca924, 0x00200012, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0x755be540
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a30: 0x00000000, 0x00000172, 0x00000172, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a40: 0x755be480, 0x755be4a0, 0x755be4c0, 0x755be4e0
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004a70: 0x28190404, 0x00305407, 0x000f001c, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b00: 0x180303fe, 0x00420003, 0x30480000, 0x003601e0
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b10: 0x003603d2, 0x007c0383, 0x0042d7f4, 0x10000d8c
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b20: 0x000000b5, 0x025a0000, 0x64a1fa81, 0x0f420048
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b30: 0x613a5441, 0x0f360038, 0x613a5281, 0x0f360038
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b40: 0x60fbf041, 0x0f7c0080, 0x98913d34, 0x655ecaba
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b50: 0xf0330000, 0x64a1fac8, 0xaf050000, 0xb5b400b4
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b60: 0xf0330000, 0x613a5478, 0xaf040000, 0x4000012f
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b70: 0xf0330000, 0x613a52b8, 0xaf040000, 0x000000ff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b80: 0x020000b4, 0x113bcb72, 0x00100800, 0x42040008
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004b90: 0x05ea4980, 0x28190404, 0x00305407, 0x000f001c
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004ba0: 0x00f00069, 0x020000b4, 0x113bcb72, 0x000f001c
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004bb0: 0x613a5601, 0x613a5c01, 0x613a5a01, 0x613a5801
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004bc0: 0x64a1fa82, 0x613a5442, 0x613a5282, 0x60fbf042
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004bd0: 0x613a5602, 0x613a5c02, 0x613a5a02, 0x613a5802
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004be0: 0x00360042, 0x007c0036, 0x0036004a, 0x00420050
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004bf0: 0xf0330000, 0x60fbf0c0, 0xaf080000, 0x0000ffff
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c10: 0x0000002a, 0x00000000, 0x00000006, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c30: 0x00000000, 0x019d8000, 0x00108000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c40: 0x00000020, 0x00000000, 0x001d0020, 0x00040020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c50: 0xd7a35675, 0x2a675aa5, 0xa63d8a36, 0x632a3232
[  221.381002] tg3 0000:06:00.1 eth2: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08002030
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005020: 0x30422000, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005040: 0x00000000, 0x00000000, 0x0800201c, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x080020ec
[  221.381002] tg3 0000:06:00.1 eth2: 0x000050a0: 0x3484dead, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000050c0: 0x00000000, 0x00000000, 0x08000088, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08002898
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005120: 0x24634521, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005140: 0x00000000, 0x00000000, 0x0800288c, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005180: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000078
[  221.381002] tg3 0000:06:00.1 eth2: 0x000051a0: 0x00831824, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000051c0: 0x00000000, 0x00000000, 0x08001a52, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005200: 0x09020010, 0x00000000, 0xc0000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005210: 0x14600027, 0x00000000, 0x3c038000, 0xc0000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005220: 0x00000b50, 0x00000000, 0x144001e5, 0x09020010
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005230: 0xc0000000, 0x400b2080, 0xc0000000, 0x14600027
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005240: 0x00000000, 0x3c038000, 0xc0010000, 0x00000b50
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005250: 0x80000000, 0x144001e5, 0x09020010, 0x41504521
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005260: 0x00000000, 0x3c038000, 0x14600027, 0x09020010
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005270: 0x3c038000, 0x144001e5, 0x09020010, 0x80000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005280: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000078
[  221.381002] tg3 0000:06:00.1 eth2: 0x000052a0: 0x00831824, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000052c0: 0x00000000, 0x00000000, 0x08001a9a, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08002050
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005320: 0x8f430000, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005340: 0x00000000, 0x00000000, 0x08000088, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08002e04
[  221.381002] tg3 0000:06:00.1 eth2: 0x000053a0: 0x00621024, 0x00000000, 0x00000000, 0x40000020
[  221.381002] tg3 0000:06:00.1 eth2: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000053c0: 0x00000000, 0x00000000, 0x08002052, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005800: 0x32000000, 0x32000000, 0x00000001, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005810: 0x73000000, 0x00000000, 0x00000001, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005820: 0xa7000000, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005860: 0x00000000, 0x00000000, 0x00000742, 0x00000742
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005880: 0x0000064a, 0x0000064a, 0x00000a36, 0x00000a36
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005890: 0x000003ad, 0x000003ad, 0x00000aa6, 0x00000aa6
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005900: 0x00000184, 0x00000184, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005980: 0x00000165, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006400: 0x00000000, 0x00000000, 0x00010891, 0xc0000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006430: 0x00000000, 0x14e4165f, 0x1786103c, 0x00020000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000064c0: 0x00000010, 0x00000004, 0x00001004, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e22
[  221.381002] tg3 0000:06:00.1 eth2: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006500: 0x01e10003, 0xfd8e8a9b, 0x00002c44, 0x00000003
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006510: 0x0007811b, 0x00058116, 0x00046113, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006550: 0x00000000, 0x02800000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006800: 0x141b0034, 0x20099082, 0x01020408, 0x0c51e788
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006810: 0x09020010, 0xffffffff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040
[  221.381002] tg3 0000:06:00.1 eth2: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x0004c728
[  221.381002] tg3 0000:06:00.1 eth2: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006900: 0xac59cd90, 0x14883791, 0x00000000, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00007000: 0x08000008, 0x00000000, 0x00000000, 0x00004af4
[  221.381002] tg3 0000:06:00.1 eth2: 0x00007010: 0x6078fa0d, 0x01c080f1, 0x00d70081, 0x03008200
[  221.381002] tg3 0000:06:00.1 eth2: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[  221.381002] tg3 0000:06:00.1 eth2: 0x00007030: 0x00060000, 0x00004af8, 0x00160030, 0x00000000
[  221.381002] tg3 0000:06:00.1 eth2: 0: Host status block [00000005:00000032:(0000:072a:0000):(0000:012d)]
[  221.381002] tg3 0000:06:00.1 eth2: 0: NAPI info [00000032:00000032:(0184:012d:01ff):0000:(0742:0000:0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 1: Host status block [00000001:000000e9:(0000:0000:0000):(0676:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 1: NAPI info [000000c2:000000c2:(0000:0000:01ff):064a:(064a:064a:0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 2: Host status block [00000001:00000073:(0a36:0000:0000):(0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 2: NAPI info [00000073:00000073:(0000:0000:01ff):0a36:(0236:0201:0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 3: Host status block [00000001:0000009d:(0000:0000:0000):(0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 3: NAPI info [00000077:00000077:(0000:0000:01ff):03ad:(03ad:03ad:0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 4: Host status block [00000001:000000a7:(0000:0000:0aa6):(0000:0000)]
[  221.381002] tg3 0000:06:00.1 eth2: 4: NAPI info [000000a7:000000a7:(0000:0000:01ff):0aa6:(02a6:0282:0000:0000)]
[  221.490166] tg3 0000:06:00.1: tg3_stop_block timed out, ofs=1400 enable_bit=2
[  221.591718] tg3 0000:06:00.1: tg3_stop_block timed out, ofs=c00 enable_bit=2
[  221.667480] tg3 0000:06:00.1 eth2: Link is down
[  223.744825] tg3 0000:06:00.1 eth2: Link is up at 100 Mbps, half duplex
[  223.746092] tg3 0000:06:00.1 eth2: Flow control is off for TX and off for RX
[  223.746092] tg3 0000:06:00.1 eth2: EEE is disabled

Comment 28 Alexander Y. Tiurin 2016-11-18 21:24:57 UTC

[  285.507432] systemd-journald[1137]: Sent WATCHDOG=1 notification.
[  311.775017] tg3 0000:06:00.1 eth2: transmit timed out, resetting
[  311.776001] tg3 0000:06:00.1 eth2: 0x00000000: 0x165f14e4, 0x00100446, 0x02000000, 0x00800010
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000010: 0xf7bc000c, 0x00000000, 0xf7bb000c, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000020: 0xf7ba000c, 0x00000000, 0x00000000, 0x1786103c
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x0000020a
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000040: 0x00000000, 0x52000000, 0xc8035001, 0x64002008
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000070: 0x000710b0, 0xffbaddff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000080: 0x165f14e4, 0x800000f8, 0x00000000, 0x000002ce
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x0000043f
[  313.030524] tg3 0000:06:00.1 eth2: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
[  313.030524] tg3 0000:06:00.1 eth2: 0x000000b0: 0x10008d81, 0x00105426, 0x0004cc22, 0x10110040
[  313.030524] tg3 0000:06:00.1 eth2: 0x000000d0: 0x0000001f, 0x00000000, 0x00000000, 0x00010000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000100: 0x13c10001, 0x00100000, 0x00100000, 0x0017f030
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000110: 0x00000000, 0x000031c0, 0x000000a0, 0x05000001
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000120: 0x00181c03, 0x08f80000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000140: 0xfd8e8a9b, 0x00002c44, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000200: 0x00000000, 0x52000000, 0x00000000, 0x8a000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000210: 0x00000000, 0x00000001, 0x00000000, 0xa8000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000220: 0x00000000, 0x34000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x00000442
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000280: 0x00000000, 0x000002ce, 0x00000000, 0x00000251
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000290: 0x00000000, 0x0000014a, 0x00000000, 0x00000511
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000300: 0x00000000, 0x000001f4, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000400: 0x18e04806, 0x00400000, 0x00001000, 0x00000900
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000410: 0x00002c44, 0xfd8e8a9b, 0x00002c44, 0xfd8e8a9b
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000420: 0x00002c44, 0xfd8e8a9b, 0x00002c44, 0xfd8e8a9b
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000430: 0x00000400, 0x00000000, 0x000000eb, 0x000005f2
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x084a0000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000460: 0x00000008, 0x00002620, 0x01ff0002, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000470: 0x00000000, 0x10000000, 0x00000000, 0x40000042
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000005c0: 0x5bbbbbe8, 0xe7d560b7, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000600: 0xffffffff, 0x00f800d1, 0x00000000, 0x00001f04
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000610: 0xffffffff, 0x00000000, 0x07c00284, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000670: 0xe43ef12c, 0xa0f4458d, 0xd216bf6c, 0x4c9eb80d
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000680: 0xd3f7d46c, 0x64bc431e, 0x39ba2e80, 0xef0aac58
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000690: 0xf252af49, 0xc218b638, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000880: 0x000b979e, 0x00000000, 0x00000000, 0x00000825
[  313.030524] tg3 0000:06:00.1 eth2: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000900: 0x0f2565d3, 0xffffffff, 0x0001839c, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000910: 0x00000000, 0xffffffff, 0x0000004e, 0x0000580c
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000920: 0x000055dd, 0x00002bc8, 0xffffffff, 0x00000006
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000930: 0x00016d4f, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x000738c1
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000970: 0x00000003, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000980: 0x000b975e, 0x00000000, 0x00000000, 0x00088b4d
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000990: 0x0000002e, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000c10: 0x00000000, 0x00000007, 0x00000000, 0x004a0000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000c80: 0x0007855e, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000ce0: 0x6028c442, 0x00000000, 0x0000004a, 0x00040028
[  313.030524] tg3 0000:06:00.1 eth2: 0x00000cf0: 0x00000000, 0x5000004e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001000: 0x00000002, 0x00000000, 0xa000b8b8, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001010: 0x51c71c81, 0x0000b8b8, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001440: 0x000001cf, 0x000001cf, 0x000001cf, 0x000001cf
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001450: 0x000001cf, 0x000001cf, 0x000001cf, 0x000001cf
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001460: 0x000001cf, 0x000001cf, 0x000001cf, 0x000001cf
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001470: 0x000001cf, 0x000001cf, 0x000001cf, 0x000001cf
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001480: 0x00001114, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001800: 0x00000016, 0x00000000, 0x000001f4, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0x75470000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001840: 0x75470000, 0x00000000, 0x00000201, 0xc0000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001850: 0x0000001f, 0x00000000, 0x00004310, 0x01f401f4
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001860: 0x03000000, 0x00000000, 0x75471f10, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002100: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002110: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002120: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002130: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002140: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002150: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002160: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002170: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002180: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002190: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021a0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021b0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021c0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021d0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021e0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000021f0: 0x000ca85e, 0x000ca85e, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002200: 0x00088b7c, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002250: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002400: 0x00010012, 0x00000000, 0x00246001, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002450: 0x00000000, 0x755b0000, 0x08001800, 0x00040000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002470: 0x00000000, 0x0000037c, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025e0: 0x00000000, 0x00000000, 0x00000002, 0x00046400
[  313.030524] tg3 0000:06:00.1 eth2: 0x000025f0: 0x00000000, 0x00000000, 0x00000002, 0x00042000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002600: 0x00000000, 0x00000000, 0x00000002, 0x00046800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002610: 0x00000000, 0x00000000, 0x00000002, 0x00042400
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002620: 0x00000000, 0x00000000, 0x00000002, 0x00046c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002630: 0x00000000, 0x00000000, 0x00000002, 0x00042800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002640: 0x00000000, 0x00000000, 0x00000002, 0x00047000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002650: 0x00000000, 0x00000000, 0x00000002, 0x00042c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002660: 0x00000000, 0x00000000, 0x00000002, 0x00047400
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002670: 0x00000000, 0x00000000, 0x00000002, 0x00043000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002680: 0x00000000, 0x00000000, 0x00000002, 0x00047800
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002690: 0x00000000, 0x00000000, 0x00000002, 0x00043400
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026a0: 0x00000000, 0x00000000, 0x00000002, 0x00047c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026b0: 0x00000000, 0x00000000, 0x00000002, 0x00043800
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026c0: 0x00000000, 0x00000000, 0x00000002, 0x00048000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026d0: 0x00000000, 0x00000000, 0x00000002, 0x00043c00
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026e0: 0x00000000, 0x00000000, 0x00000002, 0x00048400
[  313.030524] tg3 0000:06:00.1 eth2: 0x000026f0: 0x00000000, 0x00000000, 0x00000002, 0x00044000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x00000676
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002c20: 0x00000004, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x00000676
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003440: 0x00000000, 0x00000000, 0x00000000, 0x00000110
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003600: 0x00000600, 0x00170000, 0x00110000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003610: 0x00170000, 0x00000000, 0x00130000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003620: 0x00110011, 0x00000000, 0x00000000, 0x400b2080
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003630: 0x00800000, 0x87748774, 0x02c01000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003640: 0x00000000, 0x00000000, 0x00000020, 0x00000019
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003650: 0x00000171, 0x000f03ff, 0x05720000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003660: 0x00000000, 0x00000000, 0x02000000, 0x00000202
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003670: 0x00000000, 0xfeffafe3, 0x00000000, 0x00000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003680: 0x30018010, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000036a0: 0x000001a0, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000036b0: 0x0010034c, 0x07ff07ff, 0x07ff07ff, 0x01000004
[  313.030524] tg3 0000:06:00.1 eth2: 0x000036c0: 0xfffffffe, 0x00000000, 0x00000000, 0x0004ff0e
[  313.030524] tg3 0000:06:00.1 eth2: 0x000036d0: 0x000001f8, 0x00000000, 0x00000000, 0x00004d3b
[  313.030524] tg3 0000:06:00.1 eth2: 0x000036f0: 0x00000000, 0x00000000, 0x00000000, 0x00012301
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003800: 0x00000001, 0x00000000, 0x0000000e, 0x0516028b
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003810: 0x000001ff, 0x0000009f, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c00: 0x00000306, 0x00000000, 0x00000000, 0x00000048
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c30: 0x00000000, 0x00000000, 0x00000000, 0x7bc6b000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c50: 0x00000000, 0x000005fa, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003c80: 0x0000037f, 0x000002e3, 0x000001ec, 0x000005ac
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003cc0: 0x000001c7, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003d00: 0x00000000, 0x7c065000, 0x00000000, 0x7c207000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003d10: 0x00000000, 0x7c26e000, 0x00000000, 0x75582000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00003fc0: 0x000a9df5, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004000: 0x00000002, 0x00000000, 0x0011c35a, 0x001f072f
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004010: 0x00000000, 0x0023c012, 0x00000490, 0x0085ec42
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004020: 0x00016a11, 0x00000000, 0x00000010, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004040: 0x00000000, 0x00000000, 0x01081a20, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004050: 0x00000000, 0x00000000, 0x0023c010, 0x00468002
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004060: 0x00474000, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004400: 0x00000016, 0x00000000, 0x00010000, 0x0000a000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x0df018c7
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004450: 0x00039d3f, 0x003e003f, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004800: 0x380303fe, 0x00000000, 0x00000000, 0x00000120
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004810: 0x00000000, 0x0000000b, 0x057882f4, 0x00001514
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004820: 0x00000074, 0x00000000, 0xf0330000, 0x61107e48
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004830: 0x00000000, 0x00000301, 0x00000301, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004840: 0x0001da2e, 0x0001da2e, 0x000e2200, 0x74428c76
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004850: 0xa6e90047, 0x29eb05c2, 0x68676867, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004860: 0x02000074, 0x113e9403, 0x00100800, 0xea040008
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004870: 0x05ea4985, 0x003e1820, 0x003e1820, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004900: 0x28190404, 0x00305407, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x86970000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a10: 0x755bcf20, 0x008ca924, 0x00200012, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0x755bcf80
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a30: 0x00000000, 0x000001d8, 0x000001d8, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a40: 0x755bcec0, 0x755bcee0, 0x755bcf00, 0x755bcf20
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004a70: 0x28190404, 0x00305407, 0x000f001c, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b00: 0x180303fe, 0x00360003, 0x30380000, 0x003601e0
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b10: 0x003603d2, 0x00bf0383, 0x0036c2fc, 0x18002fdc
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b20: 0x000000c6, 0x025c8000, 0x60380201, 0x0f360038
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b30: 0x6028c641, 0x0f360038, 0x61106541, 0x0f360038
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b40: 0x6028d181, 0x0fbf00c8, 0x01fad6cf, 0x746d725a
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b50: 0xf0330000, 0x60380238, 0xaf040000, 0xc6b900c5
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b60: 0xf0330000, 0x6028c678, 0xaf040000, 0x400001c9
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b70: 0xf0330000, 0x61106578, 0xaf040000, 0x000000ff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b80: 0x020000b9, 0x113e9403, 0x00100800, 0xea040008
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004b90: 0x05ea4985, 0x28190404, 0x00305407, 0x000f001c
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004ba0: 0x00f000fb, 0x020000b9, 0x113e9403, 0x000f001c
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004bb0: 0x60439441, 0x6028d601, 0x649e9001, 0x6028d801
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004bc0: 0x60380202, 0x6028c642, 0x61106542, 0x6028d182
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004bd0: 0x60439040, 0x6028d602, 0x649e8c40, 0x6028d802
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004be0: 0x00360036, 0x00bf0036, 0x004205ea, 0x003605e2
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004bf0: 0xf0330000, 0x6028d248, 0xaf0c0000, 0x0000ffff
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c10: 0x0000002a, 0x00000000, 0x00000006, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c30: 0x00000000, 0x00000000, 0x00108000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c40: 0x00000020, 0x00000000, 0x001d0020, 0x00040020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c50: 0xeb2e237e, 0x0037e5ad, 0xae1ec2e3, 0x637c5252
[  313.030524] tg3 0000:06:00.1 eth2: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005000: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08002044
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005020: 0x3c034000, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005040: 0x00000000, 0x00000000, 0x08000088, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x080020ec
[  313.030524] tg3 0000:06:00.1 eth2: 0x000050a0: 0x30420001, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000050c0: 0x00000000, 0x00000000, 0x08002052, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08002898
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005120: 0x3c034150, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005140: 0x00000000, 0x00000000, 0x0800288c, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005180: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000078
[  313.030524] tg3 0000:06:00.1 eth2: 0x000051a0: 0x00831824, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000051c0: 0x00000000, 0x00000000, 0x08001a52, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005200: 0x01020000, 0x00000000, 0xc0000000, 0xc0000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005210: 0x14600027, 0xc0000000, 0x3c038000, 0x14600027
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005220: 0x08006f80, 0x00000000, 0x144001e5, 0x01020000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005230: 0x00000000, 0x400b2080, 0x00000000, 0x14600027
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005240: 0x08006f98, 0x3c038000, 0x14600027, 0x08002e44
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005250: 0xc0000000, 0x144001e5, 0x01020000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005260: 0xc0000000, 0x00000000, 0x14600027, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005270: 0x3c038000, 0x00000000, 0x00000b50, 0x80000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000078
[  313.030524] tg3 0000:06:00.1 eth2: 0x000052a0: 0x00831824, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000052c0: 0x00000000, 0x00000000, 0x08002898, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08002044
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005320: 0x27bdffe8, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005340: 0x00000000, 0x00000000, 0x08000088, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x080020ec
[  313.030524] tg3 0000:06:00.1 eth2: 0x000053a0: 0x30420001, 0x00000000, 0x00000000, 0x40000020
[  313.030524] tg3 0000:06:00.1 eth2: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000053c0: 0x00000000, 0x00000000, 0x08002052, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005800: 0x52000000, 0x52000000, 0x3b000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005810: 0x63000000, 0x00000000, 0x3e000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005820: 0xc8000000, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005860: 0x00000000, 0x00000000, 0x000006c4, 0x000006c4
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005880: 0x0000037f, 0x0000037f, 0x000002e3, 0x000002e3
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005890: 0x000001ec, 0x000001ec, 0x000005ae, 0x000005ae
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005900: 0x000001f4, 0x000001f4, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005980: 0x000001f4, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006400: 0x00000000, 0x00000000, 0x00010891, 0xc0000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006430: 0x00000000, 0x14e4165f, 0x1786103c, 0x00020000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000064c0: 0x00000010, 0x00000004, 0x00001004, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e22
[  313.030524] tg3 0000:06:00.1 eth2: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006500: 0x01e10003, 0xfd8e8a9b, 0x00002c44, 0x00000003
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006510: 0x0007811b, 0x00058116, 0x00046113, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006550: 0x00000000, 0x02800000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006800: 0x141b0034, 0x20099082, 0x01020408, 0x0562f715
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040
[  313.030524] tg3 0000:06:00.1 eth2: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x0004c728
[  313.030524] tg3 0000:06:00.1 eth2: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006900: 0xe9677f18, 0x148837a7, 0x00000000, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00007000: 0x08000008, 0x00000000, 0x00000000, 0x00004af4
[  313.030524] tg3 0000:06:00.1 eth2: 0x00007010: 0x6078fa0d, 0x01c080f1, 0x00d70081, 0x03008200
[  313.030524] tg3 0000:06:00.1 eth2: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[  313.030524] tg3 0000:06:00.1 eth2: 0x00007030: 0x00060000, 0x00004af8, 0x00160030, 0x00000000
[  313.030524] tg3 0000:06:00.1 eth2: 0: Host status block [00000001:00000052:(0000:05f9:0000):(0000:01c7)]
[  313.030524] tg3 0000:06:00.1 eth2: 0: NAPI info [00000052:00000052:(01f4:01c7:01ff):0000:(0082:0000:0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 1: Host status block [00000001:000000b0:(0000:0000:0000):(060c:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 1: NAPI info [000000b0:000000b0:(0000:0000:01ff):060c:(060c:060c:0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 2: Host status block [00000001:000000a3:(0553:0000:0000):(0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 2: NAPI info [000000a3:000000a3:(0000:0000:01ff):0553:(0553:0553:0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 3: Host status block [00000001:00000066:(0000:0000:0000):(0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 3: NAPI info [00000066:00000066:(0000:0000:01ff):043f:(043f:043f:0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 4: Host status block [00000001:0000000d:(0000:0000:081c):(0000:0000)]
[  313.030524] tg3 0000:06:00.1 eth2: 4: NAPI info [0000000d:0000000d:(0000:0000:01ff):081c:(001c:001c:0000:0000)]
[  313.140630] tg3 0000:06:00.1: tg3_stop_block timed out, ofs=1400 enable_bit=2
[  313.241957] tg3 0000:06:00.1: tg3_stop_block timed out, ofs=c00 enable_bit=2
[  313.316238] tg3 0000:06:00.1 eth2: Link is down
[  315.395006] tg3 0000:06:00.1 eth2: Link is up at 100 Mbps, half duplex
[  315.397025] tg3 0000:06:00.1 eth2: Flow control is off for TX and off for RX
[  315.397025] tg3 0000:06:00.1 eth2: EEE is disabled

Comment 29 Andrew 2017-04-26 08:12:00 UTC

I am having the same problem with  HP EliteDesk 705 G2 MINI using their built in Broadcom BCM5762 GbE NIC.
Kernel: Linux PC38858 4.10.0-20-generic #22-Ubuntu SMP Thu Apr 20 09:22:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
in Kubuntu 17.04 Also in FOG (for imaging)


01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5762 Gigabit Ethernet PCIe (rev 10)
        DeviceName: Onboard LAN
        Subsystem: Hewlett-Packard Company NetXtreme BCM5762 Gigabit Ethernet PCIe
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at d0820000 (64-bit, prefetchable) [size=64K]
        Memory at d0810000 (64-bit, prefetchable) [size=64K]
        Memory at d0800000 (64-bit, prefetchable) [size=64K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number 00-00-c8-d3-ff-a8-5f-d7
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [160] Virtual Channel
        Capabilities: [1b0] Latency Tolerance Reporting
        Capabilities: [230] Transaction Processing Hints
        Kernel driver in use: tg3
        Kernel modules: tg3


[   23.117560] ------------[ cut here ]------------
[   23.117575] WARNING: CPU: 2 PID: 0 at /build/linux-2NWldV/linux-4.10.0/net/sched/sch_generic.c:316 dev_watchdog+0x22c/0x230
[   23.117577] NETDEV WATCHDOG: eno1 (tg3): transmit queue 0 timed out
[   23.117578] Modules linked in: ccm rfcomm cmac bnep nls_iso8859_1 hp_wmi sparse_keymap edac_mce_amd edac_core kvm irqbypass arc4 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi input_leds aesni_intel snd_hda_intel aes_x86_64 iwlmvm snd_hda_codec crypto_simd glue_helper snd_hda_core cryptd snd_hwdep mac80211 snd_pcm snd_seq_midi snd_seq_midi_event fam15h_power snd_rawmidi btusb btrtl btbcm i2c_piix4 btintel k10temp bluetooth snd_seq snd_seq_device snd_timer iwlwifi cfg80211 snd shpchp soundcore mac_hid parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 amdgpu i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci tg3 drm libahci ptp pps_core wmi fjes video
[   23.117661] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.10.0-20-generic #22-Ubuntu
[   23.117663] Hardware name: HP HP EliteDesk 705 G2 MINI/805B, BIOS N26 Ver. 02.12 03/27/2017
[   23.117664] Call Trace:
[   23.117668]  <IRQ>
[   23.117674]  dump_stack+0x63/0x81
[   23.117678]  __warn+0xcb/0xf0
[   23.117680]  warn_slowpath_fmt+0x5f/0x80
[   23.117685]  dev_watchdog+0x22c/0x230
[   23.117688]  ? qdisc_rcu_free+0x50/0x50
[   23.117692]  call_timer_fn+0x35/0x140
[   23.117695]  run_timer_softirq+0x215/0x4b0
[   23.117698]  ? ktime_get+0x41/0xb0
[   23.117702]  ? lapic_next_event+0x1d/0x30
[   23.117706]  ? clockevents_program_event+0x7f/0x120
[   23.117709]  __do_softirq+0x104/0x2af
[   23.117713]  irq_exit+0xb6/0xc0
[   23.117715]  smp_apic_timer_interrupt+0x3d/0x50
[   23.117719]  apic_timer_interrupt+0x89/0x90
[   23.117724] RIP: 0010:native_safe_halt+0x6/0x10
[   23.117726] RSP: 0018:ffffb44d40cd3dc8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[   23.117729] RAX: 0000000000000000 RBX: ffff95f4f549f064 RCX: 000000000000001f
[   23.117731] RDX: ffff95f4fed00000 RSI: ffffffffb1ceb920 RDI: ffff95f4f549f064
[   23.117732] RBP: ffffb44d40cd3dc8 R08: cccccccccccccccd R09: 0000000000000018
[   23.117734] R10: 00000000000001bd R11: 000000000000048d R12: 0000000000000001
[   23.117736] R13: ffff95f4f549f000 R14: 0000000000000001 R15: ffffffffb1ceb980
[   23.117738]  </IRQ>
[   23.117743]  arch_safe_halt+0x9/0xd
[   23.117746]  acpi_safe_halt+0x1e/0x27
[   23.117749]  acpi_idle_do_entry+0x20/0x3b
[   23.117777]  acpi_idle_enter+0x1c7/0x205
[   23.117784]  cpuidle_enter_state+0xf2/0x2c0
[   23.117787]  cpuidle_enter+0x17/0x20
[   23.117815]  call_cpuidle+0x23/0x40
[   23.117818]  do_idle+0x189/0x200
[   23.117821]  cpu_startup_entry+0x71/0x80
[   23.117825]  start_secondary+0x154/0x190
[   23.117828]  start_cpu+0x14/0x14
[   23.117836] ---[ end trace f486dcc983207c48 ]---
[   23.117852] tg3 0000:01:00.0 eno1: transmit timed out, resetting
[   25.837055] tg3 0000:01:00.0 eno1: 0x00000000: 0x168714e4, 0x00100506, 0x02000010, 0x00000000                                       
[   25.837066] tg3 0000:01:00.0 eno1: 0x00000010: 0xd082000c, 0x00000000, 0xd081000c, 0x00000000                                       
[   25.837072] tg3 0000:01:00.0 eno1: 0x00000020: 0xd080000c, 0x00000000, 0x00000000, 0x805b103c                                       
[   25.837077] tg3 0000:01:00.0 eno1: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000105                                       
[   25.837083] tg3 0000:01:00.0 eno1: 0x00000040: 0x00000000, 0x3c000000, 0xc8035001, 0x16002008                                       
[   25.837087] tg3 0000:01:00.0 eno1: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000                                       
[   25.837092] tg3 0000:01:00.0 eno1: 0x00000060: 0x00000000, 0x00000000, 0xf1000298, 0x761b400f                                       
[   25.837097] tg3 0000:01:00.0 eno1: 0x00000070: 0x00071010, 0x31000200, 0x00000000, 0x00000000                                       
[   25.837101] tg3 0000:01:00.0 eno1: 0x00000080: 0x168714e4, 0xc00000d0, 0x00000000, 0x0000002c                                       
[   25.837105] tg3 0000:01:00.0 eno1: 0x00000090: 0x00000000, 0x00000041, 0x00000000, 0x0000012c                                       
[   25.837110] tg3 0000:01:00.0 eno1: 0x000000a0: 0x8005ac11, 0x00000004, 0x00000122, 0x00020010                                       
[   25.837114] tg3 0000:01:00.0 eno1: 0x000000b0: 0x10008d82, 0x00105440, 0x00475c12, 0x10110043                                       
[   25.837118] tg3 0000:01:00.0 eno1: 0x000000d0: 0x0008081f, 0x00000000, 0x00000000, 0x00010001                                       
[   25.837123] tg3 0000:01:00.0 eno1: 0x000000f0: 0x00000000, 0x05762100, 0x00000000, 0xfdfffbb7                                       
[   25.837127] tg3 0000:01:00.0 eno1: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030                                       
[   25.837131] tg3 0000:01:00.0 eno1: 0x00000110: 0x00000000, 0x00002000, 0x000000a0, 0x00000000                                       
[   25.837136] tg3 0000:01:00.0 eno1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003                                       
[   25.837140] tg3 0000:01:00.0 eno1: 0x00000140: 0xffa85fd7, 0x0000c8d3, 0x00000000, 0x00000000                                       
[   25.837144] tg3 0000:01:00.0 eno1: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000001                                       
[   25.837148] tg3 0000:01:00.0 eno1: 0x00000160: 0x1b010002, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837153] tg3 0000:01:00.0 eno1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000                                       
[   25.837157] tg3 0000:01:00.0 eno1: 0x000001b0: 0x23010018, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837161] tg3 0000:01:00.0 eno1: 0x00000200: 0x00000000, 0x3c000000, 0x00000000, 0x2d000000                                       
[   25.837165] tg3 0000:01:00.0 eno1: 0x00000210: 0x00000000, 0x1d000000, 0x00000000, 0x14000000                                       
[   25.837169] tg3 0000:01:00.0 eno1: 0x00000220: 0x00000000, 0x06000000, 0x00000000, 0x00000000                                       
[   25.837174] tg3 0000:01:00.0 eno1: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x0000012c                                       
[   25.837178] tg3 0000:01:00.0 eno1: 0x00000280: 0x00000000, 0x0000002c, 0x00000000, 0x0000001c                                       
[   25.837182] tg3 0000:01:00.0 eno1: 0x00000290: 0x00000000, 0x00000017, 0x00000000, 0x00000005                                       
[   25.837186] tg3 0000:01:00.0 eno1: 0x00000300: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837190] tg3 0000:01:00.0 eno1: 0x00000310: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837194] tg3 0000:01:00.0 eno1: 0x00000320: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837198] tg3 0000:01:00.0 eno1: 0x00000330: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837202] tg3 0000:01:00.0 eno1: 0x00000340: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837206] tg3 0000:01:00.0 eno1: 0x00000350: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837210] tg3 0000:01:00.0 eno1: 0x00000360: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837214] tg3 0000:01:00.0 eno1: 0x00000370: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837218] tg3 0000:01:00.0 eno1: 0x00000380: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837222] tg3 0000:01:00.0 eno1: 0x00000390: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837226] tg3 0000:01:00.0 eno1: 0x000003a0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837230] tg3 0000:01:00.0 eno1: 0x000003b0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837234] tg3 0000:01:00.0 eno1: 0x000003c0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837238] tg3 0000:01:00.0 eno1: 0x000003d0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837242] tg3 0000:01:00.0 eno1: 0x000003e0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837246] tg3 0000:01:00.0 eno1: 0x000003f0: 0x00000000, 0x00000041, 0x00000000, 0x00000041                                       
[   25.837250] tg3 0000:01:00.0 eno1: 0x00000400: 0x18e04804, 0x00400000, 0x00001000, 0x00009900                                       
[   25.837255] tg3 0000:01:00.0 eno1: 0x00000410: 0x0000c8d3, 0xffa85fd7, 0x0000c8d3, 0xffa85fd7                                       
[   25.837259] tg3 0000:01:00.0 eno1: 0x00000420: 0x0000c8d3, 0xffa85fd7, 0x0000c8d3, 0xffa85fd7                                       
[   25.837263] tg3 0000:01:00.0 eno1: 0x00000430: 0x00000000, 0x00000000, 0x0000037c, 0x000005f2                                       
[   25.837267] tg3 0000:01:00.0 eno1: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400                                       
[   25.837272] tg3 0000:01:00.0 eno1: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102                                       
[   25.837276] tg3 0000:01:00.0 eno1: 0x00000460: 0x00000008, 0x00002620, 0x03ff0002, 0x00000000                                       
[   25.837280] tg3 0000:01:00.0 eno1: 0x00000470: 0xa0000010, 0x00000010, 0x80000000, 0x50000000                                       
[   25.837284] tg3 0000:01:00.0 eno1: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff                                       
[   25.837288] tg3 0000:01:00.0 eno1: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000                                       
[   25.837292] tg3 0000:01:00.0 eno1: 0x00000590: 0x00901011, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837297] tg3 0000:01:00.0 eno1: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x0000000c                                       
[   25.837301] tg3 0000:01:00.0 eno1: 0x00000610: 0xffffffff, 0x00000000, 0x00000084, 0xe8000000                                       
[   25.837305] tg3 0000:01:00.0 eno1: 0x00000620: 0x00000436, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837309] tg3 0000:01:00.0 eno1: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123                                       
[   25.837313] tg3 0000:01:00.0 eno1: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123                                       
[   25.837317] tg3 0000:01:00.0 eno1: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123                                       
[   25.837321] tg3 0000:01:00.0 eno1: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123                                       
[   25.837325] tg3 0000:01:00.0 eno1: 0x00000670: 0x4807eb29, 0xcfd7552d, 0x4c32f5e7, 0xe669a038                                       
[   25.837330] tg3 0000:01:00.0 eno1: 0x00000680: 0x211f156b, 0x5fdadc1f, 0x441571b0, 0xfcc8287f                                       
[   25.837334] tg3 0000:01:00.0 eno1: 0x00000690: 0x8ae8aaf7, 0xe3c6da40, 0x00000000, 0x00000000                                       
[   25.837338] tg3 0000:01:00.0 eno1: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000                                       
[   25.837342] tg3 0000:01:00.0 eno1: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000                                       
[   25.837347] tg3 0000:01:00.0 eno1: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000                                       
[   25.837351] tg3 0000:01:00.0 eno1: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000                                       
[   25.837355] tg3 0000:01:00.0 eno1: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837359] tg3 0000:01:00.0 eno1: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837363] tg3 0000:01:00.0 eno1: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837367] tg3 0000:01:00.0 eno1: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000                                       
[   25.837374] tg3 0000:01:00.0 eno1: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837378] tg3 0000:01:00.0 eno1: 0x00000900: 0x00004123, 0xffffffff, 0x00000000, 0x00000000                                       
[   25.837382] tg3 0000:01:00.0 eno1: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000                                       
[   25.837386] tg3 0000:01:00.0 eno1: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000                                       
[   25.837390] tg3 0000:01:00.0 eno1: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837394] tg3 0000:01:00.0 eno1: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837417] tg3 0000:01:00.0 eno1: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff                                       
[   25.837421] tg3 0000:01:00.0 eno1: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000011                                       
[   25.837425] tg3 0000:01:00.0 eno1: 0x00000970: 0x00000028, 0x00000007, 0x00000000, 0x00000000                                       
[   25.837475] tg3 0000:01:00.0 eno1: 0x00000980: 0x0000f021, 0x00000000, 0x00000000, 0x00000023                                       
[   25.837479] tg3 0000:01:00.0 eno1: 0x00000990: 0x00000074, 0x00000096, 0x00000000, 0x00000000                                       
[   25.837485] tg3 0000:01:00.0 eno1: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001                                       
[   25.837489] tg3 0000:01:00.0 eno1: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x03df0000                                       
[   25.837498] tg3 0000:01:00.0 eno1: 0x00000c80: 0x00000040, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837503] tg3 0000:01:00.0 eno1: 0x00000ce0: 0x32cf9002, 0x00000002, 0x000003df, 0x00041028                                       
[   25.837507] tg3 0000:01:00.0 eno1: 0x00000cf0: 0x00000000, 0x50000041, 0x00000000, 0x00000000                                       
[   25.837513] tg3 0000:01:00.0 eno1: 0x00001000: 0x00000002, 0x00000000, 0xa000b8c0, 0x00000000                                       
[   25.837520] tg3 0000:01:00.0 eno1: 0x00001010: 0x00410411, 0x0000b8c0, 0x00000000, 0x00000000                                       
[   25.837526] tg3 0000:01:00.0 eno1: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000001                                       
[   25.837531] tg3 0000:01:00.0 eno1: 0x00001440: 0x00000041, 0x00000041, 0x00000041, 0x00000041                                       
[   25.837536] tg3 0000:01:00.0 eno1: 0x00001450: 0x00000041, 0x00000041, 0x00000041, 0x00000041                                       
[   25.837540] tg3 0000:01:00.0 eno1: 0x00001460: 0x00000041, 0x00000041, 0x00000041, 0x00000041                                       
[   25.837545] tg3 0000:01:00.0 eno1: 0x00001470: 0x00000041, 0x00000041, 0x00000041, 0x00000041                                       
[   25.837550] tg3 0000:01:00.0 eno1: 0x00001800: 0x00000016, 0x00000000, 0x00000041, 0x00000000                                       
[   25.837554] tg3 0000:01:00.0 eno1: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010                                       
[   25.837559] tg3 0000:01:00.0 eno1: 0x00001850: 0x0000001f, 0x00000002, 0x00004000, 0x00410041                                       
[   25.837564] tg3 0000:01:00.0 eno1: 0x00001860: 0x01000000, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837569] tg3 0000:01:00.0 eno1: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837574] tg3 0000:01:00.0 eno1: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837579] tg3 0000:01:00.0 eno1: 0x00002010: 0x00000181, 0x00000001, 0x007bfffd, 0x00000000                                       
[   25.837590] tg3 0000:01:00.0 eno1: 0x00002100: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837602] tg3 0000:01:00.0 eno1: 0x00002110: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837613] tg3 0000:01:00.0 eno1: 0x00002120: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837624] tg3 0000:01:00.0 eno1: 0x00002130: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837634] tg3 0000:01:00.0 eno1: 0x00002140: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837645] tg3 0000:01:00.0 eno1: 0x00002150: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837655] tg3 0000:01:00.0 eno1: 0x00002160: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837666] tg3 0000:01:00.0 eno1: 0x00002170: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837676] tg3 0000:01:00.0 eno1: 0x00002180: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837686] tg3 0000:01:00.0 eno1: 0x00002190: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837697] tg3 0000:01:00.0 eno1: 0x000021a0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837707] tg3 0000:01:00.0 eno1: 0x000021b0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837717] tg3 0000:01:00.0 eno1: 0x000021c0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837728] tg3 0000:01:00.0 eno1: 0x000021d0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837738] tg3 0000:01:00.0 eno1: 0x000021e0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837749] tg3 0000:01:00.0 eno1: 0x000021f0: 0x000c984c, 0x000e6331, 0x0000005f, 0x00000000                                       
[   25.837760] tg3 0000:01:00.0 eno1: 0x00002200: 0x00000105, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837771] tg3 0000:01:00.0 eno1: 0x00002250: 0x0000007b, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837781] tg3 0000:01:00.0 eno1: 0x00002400: 0x00010012, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837792] tg3 0000:01:00.0 eno1: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000                                       
[   25.837802] tg3 0000:01:00.0 eno1: 0x00002450: 0x00000002, 0x30bc0000, 0x08001800, 0x00006000                                       
[   25.837813] tg3 0000:01:00.0 eno1: 0x00002470: 0x00000000, 0x00000070, 0x00000000, 0x00000000                                       
[   25.837824] tg3 0000:01:00.0 eno1: 0x000024c0: 0x02000101, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837836] tg3 0000:01:00.0 eno1: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837848] tg3 0000:01:00.0 eno1: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x000000e5                                       
[   25.837858] tg3 0000:01:00.0 eno1: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c                                       
[   25.837869] tg3 0000:01:00.0 eno1: 0x00002c20: 0x00020000, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837874] tg3 0000:01:00.0 eno1: 0x00002d00: 0x00000080, 0x00000040, 0x00000000, 0x00000000                                       
[   25.837880] tg3 0000:01:00.0 eno1: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x000000e5                                       
[   25.837885] tg3 0000:01:00.0 eno1: 0x00003600: 0x00034400, 0x00130000, 0x00110000, 0x00000000                                       
[   25.837908] tg3 0000:01:00.0 eno1: 0x00003610: 0x00131100, 0x00130000, 0x00130000, 0x00000000                                       
[   25.837913] tg3 0000:01:00.0 eno1: 0x00003620: 0x80110011, 0x00000000, 0x00000004, 0x3c8a2080                                       
[   25.837936] tg3 0000:01:00.0 eno1: 0x00003630: 0x00110000, 0x80008000, 0x00001080, 0x00000000                                       
[   25.837941] tg3 0000:01:00.0 eno1: 0x00003640: 0x0000000a, 0x00400000, 0x00000020, 0x00000019                                       
[   25.837946] tg3 0000:01:00.0 eno1: 0x00003650: 0x00000171, 0x00000bff, 0x05762100, 0x00000000                                       
[   25.837951] tg3 0000:01:00.0 eno1: 0x00003660: 0x00000000, 0x00000000, 0x000400a3, 0x00000202                                       
[   25.837955] tg3 0000:01:00.0 eno1: 0x00003670: 0x0000002a, 0xfeff7fe3, 0x0000000a, 0x00000000                                       
[   25.837960] tg3 0000:01:00.0 eno1: 0x000036b0: 0x0010034c, 0x07ff07ff, 0x07ff07ff, 0x01000004                                       
[   25.837965] tg3 0000:01:00.0 eno1: 0x000036c0: 0xffffffff, 0x00000000, 0x00000000, 0x0000037a                                       
[   25.837971] tg3 0000:01:00.0 eno1: 0x000036d0: 0x00000384, 0x00000000, 0x00000000, 0x00000000                                       
[   25.837976] tg3 0000:01:00.0 eno1: 0x000036e0: 0x80000b19, 0xa0800799, 0x00500799, 0x00000000                                       
[   25.837981] tg3 0000:01:00.0 eno1: 0x000036f0: 0x19090900, 0x13180600, 0x00000301, 0x00000001                                       
[   25.837986] tg3 0000:01:00.0 eno1: 0x00003700: 0x00000000, 0x00000000, 0x00000000, 0x00001388                                       
[   25.837991] tg3 0000:01:00.0 eno1: 0x00003710: 0x87fffffd, 0x00000000, 0x00000000, 0x000010dc                                       
[   25.837995] tg3 0000:01:00.0 eno1: 0x00003720: 0x00000000, 0x00002040, 0x88006000, 0xa0002000                                       
[   25.838000] tg3 0000:01:00.0 eno1: 0x00003750: 0x00000000, 0x00000000, 0x0000080a, 0x00000000                                       
[   25.838004] tg3 0000:01:00.0 eno1: 0x00003780: 0x0000b7f9, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838010] tg3 0000:01:00.0 eno1: 0x00003c00: 0x00000306, 0x00000000, 0x00000000, 0x00000048                                       
[   25.838014] tg3 0000:01:00.0 eno1: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000                                       
[   25.838018] tg3 0000:01:00.0 eno1: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000                                       
[   25.838023] tg3 0000:01:00.0 eno1: 0x00003c30: 0x00000000, 0x00000000, 0x00000002, 0x32bca000                                       
[   25.838027] tg3 0000:01:00.0 eno1: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000                                       
[   25.838031] tg3 0000:01:00.0 eno1: 0x00003c50: 0x00000000, 0x00000065, 0x00000000, 0x00000000                                       
[   25.838036] tg3 0000:01:00.0 eno1: 0x00003c80: 0x0000002c, 0x0000001d, 0x00000017, 0x00000005                                       
[   25.838041] tg3 0000:01:00.0 eno1: 0x00003cc0: 0x00000041, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838046] tg3 0000:01:00.0 eno1: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000                                       
[   25.838051] tg3 0000:01:00.0 eno1: 0x00003d00: 0x00000002, 0x32bf3000, 0x00000002, 0x316da000                                       
[   25.838056] tg3 0000:01:00.0 eno1: 0x00003d10: 0x00000002, 0x2d277000, 0x00000002, 0x2da27000                                       
[   25.838061] tg3 0000:01:00.0 eno1: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000                                       
[   25.838069] tg3 0000:01:00.0 eno1: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000                                       
[   25.838074] tg3 0000:01:00.0 eno1: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000                                       
[   25.838079] tg3 0000:01:00.0 eno1: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000                                       
[   25.838083] tg3 0000:01:00.0 eno1: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000                                       
[   25.838088] tg3 0000:01:00.0 eno1: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000                                       
[   25.838093] tg3 0000:01:00.0 eno1: 0x00004000: 0x00000002, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838099] tg3 0000:01:00.0 eno1: 0x00004010: 0x00000000, 0x00331012, 0x002221f0, 0x01000222                                       
[   25.838104] tg3 0000:01:00.0 eno1: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050                                       
[   25.838109] tg3 0000:01:00.0 eno1: 0x00004030: 0x00000000, 0x01080020, 0x00224010, 0x00000002                                       
[   25.838113] tg3 0000:01:00.0 eno1: 0x00004040: 0x00400000, 0x00000000, 0x00000010, 0x004b8062                                       
[   25.838119] tg3 0000:01:00.0 eno1: 0x00004400: 0x00000016, 0x00000000, 0x00010000, 0x0000a000                                       
[   25.838124] tg3 0000:01:00.0 eno1: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000                                       
[   25.838128] tg3 0000:01:00.0 eno1: 0x00004420: 0x000000fd, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838133] tg3 0000:01:00.0 eno1: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x0b8300c2                                       
[   25.838138] tg3 0000:01:00.0 eno1: 0x00004450: 0x02300228, 0x001b0133, 0x00000000, 0x00000000                                       
[   25.838143] tg3 0000:01:00.0 eno1: 0x00004700: 0x00030002, 0x00100000, 0x00100010, 0x00660000                                       
[   25.838148] tg3 0000:01:00.0 eno1: 0x00004710: 0x2e3ac3e0, 0x00062c8c, 0x00100011, 0x00000000                                       
[   25.838159] tg3 0000:01:00.0 eno1: 0x00004720: 0x00000000, 0x00000000, 0xf02c0000, 0x2e3ac410                                       
[   25.838170] tg3 0000:01:00.0 eno1: 0x00004770: 0x000c0404, 0x00000002, 0x00000010, 0x00000000                                       
[   25.838182] tg3 0000:01:00.0 eno1: 0x00004800: 0x380303fe, 0x00000000, 0x00000000, 0x00000100                                       
[   25.838193] tg3 0000:01:00.0 eno1: 0x00004810: 0x00000000, 0x0000000c, 0x00009f80, 0x00000000                                       
[   25.838204] tg3 0000:01:00.0 eno1: 0x00004820: 0x00000000, 0x00000000, 0x00080000, 0x00000000                                       
[   25.838215] tg3 0000:01:00.0 eno1: 0x00004860: 0x00000000, 0x1130626f, 0x001ff800, 0xdf000008                                       
[   25.838226] tg3 0000:01:00.0 eno1: 0x00004870: 0x05ea0083, 0x003e1820, 0x003e1820, 0x00000000                                       
[   25.838239] tg3 0000:01:00.0 eno1: 0x00004890: 0x280c0c04, 0x00305400, 0x00000000, 0x00000000                                       
[   25.838251] tg3 0000:01:00.0 eno1: 0x000048a0: 0x000f0010, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838262] tg3 0000:01:00.0 eno1: 0x00004900: 0x18030002, 0x00000003, 0x30000000, 0x00000120                                       
[   25.838272] tg3 0000:01:00.0 eno1: 0x00004910: 0x00000040, 0x00000003, 0x0000df94, 0x00000008                                       
[   25.838285] tg3 0000:01:00.0 eno1: 0x00004920: 0x000000c1, 0x025c0000, 0x32cf9001, 0x0f0003e8                                       
[   25.838297] tg3 0000:01:00.0 eno1: 0x00004930: 0x344a2009, 0x0f0000f0, 0x347ad009, 0x0f000388                                       
[   25.838309] tg3 0000:01:00.0 eno1: 0x00004940: 0x347ac809, 0x0f000358, 0x81810303, 0xe1e1b7b7                                       
[   25.838321] tg3 0000:01:00.0 eno1: 0x00004950: 0xf03b0000, 0x32cf93e8, 0xaf000000, 0xc1b800c0                                       
[   25.838332] tg3 0000:01:00.0 eno1: 0x00004960: 0x00000000, 0x344a20f8, 0xaf000000, 0x40000041                                       
[   25.838342] tg3 0000:01:00.0 eno1: 0x00004970: 0x00028202, 0x00205400, 0x0000001c, 0x000000ff                                       
[   25.838354] tg3 0000:01:00.0 eno1: 0x00004980: 0x000000c1, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838365] tg3 0000:01:00.0 eno1: 0x00004990: 0x00000000, 0x00000000, 0x347ad390, 0xaf000000                                       
[   25.838380] tg3 0000:01:00.0 eno1: 0x000049a0: 0x00f00049, 0x000000c1, 0x00000000, 0x00000000                                       
[   25.838398] tg3 0000:01:00.0 eno1: 0x000049b0: 0x344a1c01, 0x33b83a01, 0x33b82801, 0x344a18e9                                       
[   25.838409] tg3 0000:01:00.0 eno1: 0x000049c0: 0x32cf9002, 0x344a200a, 0x347ad00a, 0x347ac80a                                       
[   25.838421] tg3 0000:01:00.0 eno1: 0x000049d0: 0x344a1c02, 0x33b83a02, 0x33b82802, 0x344a18ee                                       
[   25.838431] tg3 0000:01:00.0 eno1: 0x000049f0: 0xf03b0000, 0x347acb60, 0xaf000000, 0x0000ffff                                       
[   25.838438] tg3 0000:01:00.0 eno1: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000001                                       
[   25.838443] tg3 0000:01:00.0 eno1: 0x00004c10: 0x00000009, 0x0000000a, 0x00000003, 0x00000023                                       
[   25.838448] tg3 0000:01:00.0 eno1: 0x00004c20: 0x00000000, 0x000002ce, 0x00000078, 0x00000006                                       
[   25.838453] tg3 0000:01:00.0 eno1: 0x00004c30: 0x00000025, 0x0000b178, 0x00000091, 0x00000014                                       
[   25.838457] tg3 0000:01:00.0 eno1: 0x00004c40: 0x800d8068, 0x001d0000, 0x00050017, 0x001d002b                                       
[   25.838462] tg3 0000:01:00.0 eno1: 0x00004c50: 0x00040016, 0x00650fff, 0x40000000, 0x55663d3d                                       
[   25.838467] tg3 0000:01:00.0 eno1: 0x00004c60: 0x03e80020, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838472] tg3 0000:01:00.0 eno1: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838477] tg3 0000:01:00.0 eno1: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x080002f8                                       
[   25.838482] tg3 0000:01:00.0 eno1: 0x00005020: 0x8e24362c, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838487] tg3 0000:01:00.0 eno1: 0x00005030: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838492] tg3 0000:01:00.0 eno1: 0x00005040: 0x00000000, 0x00000000, 0x08001514, 0x00000000                                       
[   25.838496] tg3 0000:01:00.0 eno1: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838501] tg3 0000:01:00.0 eno1: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08001088                                       
[   25.838506] tg3 0000:01:00.0 eno1: 0x000050a0: 0x00621024, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838511] tg3 0000:01:00.0 eno1: 0x000050b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838515] tg3 0000:01:00.0 eno1: 0x000050c0: 0x00000000, 0x00000000, 0x08000300, 0x00000000                                       
[   25.838520] tg3 0000:01:00.0 eno1: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838525] tg3 0000:01:00.0 eno1: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08001c6c                                       
[   25.838530] tg3 0000:01:00.0 eno1: 0x00005120: 0x27bd0028, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838535] tg3 0000:01:00.0 eno1: 0x00005130: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838539] tg3 0000:01:00.0 eno1: 0x00005140: 0x00000000, 0x00000000, 0x080012b8, 0x00000000                                       
[   25.838544] tg3 0000:01:00.0 eno1: 0x00005180: 0x00009800, 0x80000000, 0x00000000, 0x00000000                                       
[   25.838549] tg3 0000:01:00.0 eno1: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000e7c                                       
[   25.838554] tg3 0000:01:00.0 eno1: 0x000051a0: 0x30420fff, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838559] tg3 0000:01:00.0 eno1: 0x000051b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838564] tg3 0000:01:00.0 eno1: 0x000051c0: 0x00000000, 0x00000000, 0x08000e20, 0x00000000                                       
[   25.838569] tg3 0000:01:00.0 eno1: 0x00005200: 0x00000000, 0x00000000, 0x14400016, 0x80000000                                       
[   25.838574] tg3 0000:01:00.0 eno1: 0x00005210: 0x14830018, 0x3c040200, 0xc0010000, 0x08007f58                                       
[   25.838579] tg3 0000:01:00.0 eno1: 0x00005220: 0x08007f80, 0x00000000, 0x00000000, 0x14400016                                       
[   25.838584] tg3 0000:01:00.0 eno1: 0x00005230: 0x3c038000, 0xc0000000, 0x00000000, 0xc0000000                                       
[   25.838589] tg3 0000:01:00.0 eno1: 0x00005240: 0x00000000, 0x01000010, 0x03608821, 0x00000000                                       
[   25.838594] tg3 0000:01:00.0 eno1: 0x00005250: 0x00080000, 0xc0010000, 0xc0010000, 0x00000000                                       
[   25.838599] tg3 0000:01:00.0 eno1: 0x00005260: 0x00831824, 0x00000000, 0x01000010, 0x03608821                                       
[   25.838603] tg3 0000:01:00.0 eno1: 0x00005270: 0x00000000, 0x08007f60, 0x05762100, 0xc0010000                                       
[   25.838608] tg3 0000:01:00.0 eno1: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838613] tg3 0000:01:00.0 eno1: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000e3c                                       
[   25.838618] tg3 0000:01:00.0 eno1: 0x000052a0: 0x8fb00014, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838623] tg3 0000:01:00.0 eno1: 0x000052b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838627] tg3 0000:01:00.0 eno1: 0x000052c0: 0x00000000, 0x00000000, 0x08001000, 0x00000000                                       
[   25.838632] tg3 0000:01:00.0 eno1: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838637] tg3 0000:01:00.0 eno1: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08000d00                                       
[   25.838642] tg3 0000:01:00.0 eno1: 0x00005320: 0x3c034000, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838651] tg3 0000:01:00.0 eno1: 0x00005330: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838661] tg3 0000:01:00.0 eno1: 0x00005340: 0x00000000, 0x00000000, 0x08000110, 0x00000000                                       
[   25.838674] tg3 0000:01:00.0 eno1: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000                                       
[   25.838684] tg3 0000:01:00.0 eno1: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x080002f4                                       
[   25.838696] tg3 0000:01:00.0 eno1: 0x000053a0: 0x14400013, 0x00000000, 0x00000000, 0x40000020                                       
[   25.838714] tg3 0000:01:00.0 eno1: 0x000053b0: 0x00000000, 0x00000025, 0x00000000, 0x00000000                                       
[   25.838726] tg3 0000:01:00.0 eno1: 0x000053c0: 0x00000000, 0x00000000, 0x08001ce8, 0x00000000                                       
[   25.838736] tg3 0000:01:00.0 eno1: 0x00005800: 0x00000000, 0x3c000000, 0x00000000, 0x2d000000                                       
[   25.838747] tg3 0000:01:00.0 eno1: 0x00005810: 0x00000000, 0x1d000000, 0x00000000, 0x14000000                                       
[   25.838759] tg3 0000:01:00.0 eno1: 0x00005820: 0x00000000, 0x06000000, 0x00000000, 0x00000000                                       
[   25.838771] tg3 0000:01:00.0 eno1: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x0000012c                                       
[   25.838783] tg3 0000:01:00.0 eno1: 0x00005880: 0x00000000, 0x0000002c, 0x00000000, 0x0000001c                                       
[   25.838796] tg3 0000:01:00.0 eno1: 0x00005890: 0x00000000, 0x00000017, 0x00000000, 0x00000005                                       
[   25.838807] tg3 0000:01:00.0 eno1: 0x00005900: 0x00000000, 0x00000041, 0x00000000, 0x00000000                                       
[   25.838818] tg3 0000:01:00.0 eno1: 0x00005980: 0x00000041, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838828] tg3 0000:01:00.0 eno1: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000                                       
[   25.838840] tg3 0000:01:00.0 eno1: 0x00006000: 0x20010082, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838872] tg3 0000:01:00.0 eno1: 0x00006400: 0x00000000, 0x00000000, 0x00010091, 0xc0000000                                       
[   25.838899] tg3 0000:01:00.0 eno1: 0x00006410: 0x05000016, 0x05000016, 0x00000000, 0x00000000                                       
[   25.838909] tg3 0000:01:00.0 eno1: 0x00006430: 0x00000000, 0x14e41687, 0x805b103c, 0x10020000                                       
[   25.838920] tg3 0000:01:00.0 eno1: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000020                                       
[   25.838931] tg3 0000:01:00.0 eno1: 0x000064c0: 0x00000005, 0x00000004, 0x00000122, 0x00000000                                       
[   25.838942] tg3 0000:01:00.0 eno1: 0x000064d0: 0x00000040, 0x10008d82, 0x00000000, 0x00d75e12                                       
[   25.838952] tg3 0000:01:00.0 eno1: 0x000064e0: 0x00000031, 0x0008001f, 0x00000000, 0x00000000                                       
[   25.838959] tg3 0000:01:00.0 eno1: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000                                       
[   25.838964] tg3 0000:01:00.0 eno1: 0x00006500: 0x03e10003, 0xffa85fd7, 0x0000c8d3, 0x00000003                                       
[   25.838969] tg3 0000:01:00.0 eno1: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000                                       
[   25.838974] tg3 0000:01:00.0 eno1: 0x00006530: 0x00000001, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838979] tg3 0000:01:00.0 eno1: 0x00006540: 0x0028081f, 0x0001001e, 0x00000000, 0x00000000                                       
[   25.838983] tg3 0000:01:00.0 eno1: 0x00006550: 0x00000001, 0x02800000, 0x0000000f, 0x00000000                                       
[   25.838988] tg3 0000:01:00.0 eno1: 0x00006560: 0x0000000f, 0x00000000, 0x00000000, 0x00000000                                       
[   25.838993] tg3 0000:01:00.0 eno1: 0x000065f0: 0x00000000, 0x00000059, 0x00000000, 0x00000000                                       
[   25.838998] tg3 0000:01:00.0 eno1: 0x00006800: 0x141b0034, 0x20081082, 0x00029118, 0x012eebf8                                       
[   25.839003] tg3 0000:01:00.0 eno1: 0x00006810: 0x01000010, 0xffffffff, 0x00000000, 0x004000f0                                       
[   25.839008] tg3 0000:01:00.0 eno1: 0x00006880: 0x77fff020, 0x00000040, 0x00801687, 0x00000000                                       
[   25.839012] tg3 0000:01:00.0 eno1: 0x00006890: 0x00800000, 0x00000000, 0x00000000, 0x00000000                                       
[   25.839017] tg3 0000:01:00.0 eno1: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000                                       
[   25.839021] tg3 0000:01:00.0 eno1: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000                                       
[   25.839026] tg3 0000:01:00.0 eno1: 0x000068c0: 0x00000044, 0x00000000, 0x00000000, 0x00000000                                       
[   25.839030] tg3 0000:01:00.0 eno1: 0x000068f0: 0x00000000, 0x00000000, 0x00000000, 0x04444444                                       
[   25.839035] tg3 0000:01:00.0 eno1: 0x00006900: 0xfed64aa0, 0x14b8aa36, 0x00000000, 0x00000000                                       
[   25.839040] tg3 0000:01:00.0 eno1: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000                                       
[   25.839045] tg3 0000:01:00.0 eno1: 0x00006c00: 0x168714e4, 0x00100506, 0x02000010, 0x00000000                                       
[   25.839053] tg3 0000:01:00.0 eno1: 0x00006c10: 0xd082000c, 0x00000000, 0xd081000c, 0x00000000                                       
[   25.839057] tg3 0000:01:00.0 eno1: 0x00006c20: 0xd080000c, 0x00000000, 0x00000000, 0x805b103c                                       
[   25.839062] tg3 0000:01:00.0 eno1: 0x00006c30: 0x00000000, 0x00000048, 0x00000000, 0x00000105                                       
[   25.839067] tg3 0000:01:00.0 eno1: 0x00006c40: 0x00000000, 0x00000000, 0xc8035001, 0x16002008                                       
[   25.839072] tg3 0000:01:00.0 eno1: 0x00006c50: 0x00005803, 0x00000000, 0x0086a005, 0x00000000                                       
[   25.839077] tg3 0000:01:00.0 eno1: 0x00006c60: 0x00000000, 0x00000000, 0xf1000298, 0x761b400f                                       
[   25.839081] tg3 0000:01:00.0 eno1: 0x00006c70: 0x00071010, 0x31000200, 0x00000000, 0x00000000                                       
[   25.839086] tg3 0000:01:00.0 eno1: 0x00006ca0: 0x8005ac11, 0x00000004, 0x00000122, 0x00020010                                       
[   25.839091] tg3 0000:01:00.0 eno1: 0x00006cb0: 0x10008d82, 0x00105440, 0x00475c12, 0x10110043                                       
[   25.839096] tg3 0000:01:00.0 eno1: 0x00006cd0: 0x0008081f, 0x00000000, 0x00000000, 0x00010001                                       
[   25.839101] tg3 0000:01:00.0 eno1: 0x00006cf0: 0x00000000, 0x05762100, 0x00000000, 0x00000000                                       
[   25.839106] tg3 0000:01:00.0 eno1: 0x00006d00: 0x13c10001, 0x00000000, 0x00000000, 0x00062030                                       
[   25.839110] tg3 0000:01:00.0 eno1: 0x00006d10: 0x00000000, 0x00002000, 0x000000a0, 0x00000000                                       
[   25.839115] tg3 0000:01:00.0 eno1: 0x00006d30: 0x00000000, 0x00000000, 0x00000000, 0x15010003                                       
[   25.839120] tg3 0000:01:00.0 eno1: 0x00006d40: 0xffa85fd7, 0x0000c8d3, 0x00000000, 0x00000000                                       
[   25.839125] tg3 0000:01:00.0 eno1: 0x00006d50: 0x16010004, 0x00000000, 0x00078116, 0x00000001                                       
[   25.839129] tg3 0000:01:00.0 eno1: 0x00006d60: 0x1b010002, 0x00000000, 0x00000000, 0x00000000                                       
[   25.839134] tg3 0000:01:00.0 eno1: 0x00006d70: 0x00000000, 0x800000ff, 0x00000000, 0x00000000                                       
[   25.839143] tg3 0000:01:00.0 eno1: 0x00006db0: 0x23010018, 0x00000000, 0x00000000, 0x00000000                                       
[   25.839154] tg3 0000:01:00.0 eno1: 0x00006e30: 0x00010017, 0x00050403, 0x00000000, 0x00000000                                       
[   25.839164] tg3 0000:01:00.0 eno1: 0x00007000: 0x08000008, 0x00000000, 0x00000000, 0x000038d4                                       
[   25.839175] tg3 0000:01:00.0 eno1: 0x00007010: 0x9928504d, 0x00808083, 0x80050020, 0x03000200                                       
[   25.839186] tg3 0000:01:00.0 eno1: 0x00007020: 0x00000000, 0x00000008, 0x00000406, 0x10004000                                       
[   25.839197] tg3 0000:01:00.0 eno1: 0x00007030: 0x000e0000, 0x000038d8, 0x00430030, 0x80000000                                       
[   25.839209] tg3 0000:01:00.0 eno1: 0x00007500: 0x00000000, 0x00000000, 0x0000008b, 0x00000000                                       
[   25.839226] tg3 0000:01:00.0 eno1: 0: Host status block [00000001:0000003c:(0000:0061:0000):(0000:003b)]                            
[   25.839242] tg3 0000:01:00.0 eno1: 0: NAPI info [0000003c:0000003c:(0041:003b:01ff):0000:(012c:0000:0000:0000)]                     
[   25.839253] tg3 0000:01:00.0 eno1: 1: Host status block [00000001:0000002d:(0000:0000:0000):(002c:0000)]
[   25.839266] tg3 0000:01:00.0 eno1: 1: NAPI info [0000002d:0000002d:(0000:0000:01ff):002c:(002c:002c:0000:0000)]
[   25.839277] tg3 0000:01:00.0 eno1: 2: Host status block [00000001:0000001d:(001c:0000:0000):(0000:0000)]
[   25.839289] tg3 0000:01:00.0 eno1: 2: NAPI info [0000001d:0000001d:(0000:0000:01ff):001c:(001c:001c:0000:0000)]
[   25.839302] tg3 0000:01:00.0 eno1: 3: Host status block [00000001:00000014:(0000:0000:0000):(0000:0000)]
[   25.839315] tg3 0000:01:00.0 eno1: 3: NAPI info [00000014:00000014:(0000:0000:01ff):0017:(0017:0017:0000:0000)]
[   25.839327] tg3 0000:01:00.0 eno1: 4: Host status block [00000001:00000006:(0000:0000:0005):(0000:0000)]
[   25.839338] tg3 0000:01:00.0 eno1: 4: NAPI info [00000006:00000006:(0000:0000:01ff):0005:(0005:0005:0000:0000)]
[   25.963516] tg3 0000:01:00.0: tg3_stop_block timed out, ofs=4c00 enable_bit=2
[   26.128334] tg3 0000:01:00.0 eno1: Link is down

Comment 30 Krzysztof Nowicki 2017-04-26 09:54:21 UTC

Same here on a HP EliteBook 745 G3 (AMD A10-8700B CPU), kernel 4.9.5

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5762 Gigabit Ethernet PCIe (rev 10)
        Subsystem: Hewlett-Packard Company NetXtreme BCM5762 Gigabit Ethernet PCIe
        Flags: bus master, fast devsel, latency 0, IRQ 31
        Memory at d0820000 (64-bit, prefetchable) [size=64K]
        Memory at d0810000 (64-bit, prefetchable) [size=64K]
        Memory at d0800000 (64-bit, prefetchable) [size=64K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number 00-00-ec-8e-b5-xx-xx-xx
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [160] Virtual Channel
        Capabilities: [1b0] Latency Tolerance Reporting
        Capabilities: [230] Transaction Processing Hints
        Kernel driver in use: tg3

I'll skip pasting the registry dump, as it's almost identical, but before the kernel warning I'm getting some AMD-Vi errors:

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff100 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff280 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff300 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff340 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff140 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff1c0 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff200 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff240 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff4c0 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff380 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff2c0 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff180 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff440 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff480 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff400 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff3c0 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff500 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff5c0 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff540 flags=0x0000]
AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0001 address=0x00000001fffff580 flags=0x0000]

I have tried to disable AMD IOMMU via kernel command line, but that doesn't solve the problem.

One thing worth noting is that this happens rarely during system idle, but quite often when the system is under heavy load. In my case it is very easy reproducible when compiling a large package with distcc using Gentoo portage with MAKEOPTS set to -j12.

Similar issues (IOMMU errors followed by a kernel warning about queue timeouts) with my integrated Intel Wireless 7265 NIC, which leads me to suspicions that this is somehow related to the AMD platform rather than the device itself (I've been using both Broadcom NetXtreme and Intel Wifi on Intel-based systems without any issues).

Comment 31 egorenar-dev 2019-10-27 10:45:57 UTC

Hi, i'm having a similar issue on iMac5k with Void Linux

Linux 5.3.7_1 #1 SMP PREEMPT

04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe
	Flags: bus master, fast devsel, latency 0, IRQ 17
	Memory at 92510000 (64-bit, prefetchable) [size=64K]
	Memory at 92500000 (64-bit, prefetchable) [size=64K]
	Expansion ROM at 92520000 [disabled] [size=2K]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [13c] Device Serial Number ...
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Capabilities: [1b0] Latency Tolerance Reporting
	Kernel driver in use: tg3
	Kernel modules: tg3



[ 7424.223510] ------------[ cut here ]------------
[ 7424.223521] NETDEV WATCHDOG: enp4s0f0 (tg3): transmit queue 0 timed out
[ 7424.223556] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x26f/0x280
[ 7424.223560] Modules linked in: md4 cmac nls_utf8 cifs gcm libarc4 fscache xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_tables x_tables br_netfilter bridge cfg80211 8021q garp mrp stp llc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp iTCO_wdt iTCO_vendor_support mei_hdcp kvm_intel uvcvideo videobuf2_vmalloc videobuf2_memops snd_hda_codec_hdmi kvm videobuf2_v4l2 videobuf2_common irqbypass snd_hda_intel videodev joydev intel_cstate snd_hda_codec pcspkr i2c_i801 mc intel_rapl_perf input_leds usbmouse snd_hda_core acpi_als kfifo_buf snd_hwdep evdev mei_me industrialio thunderbolt tg3 mac_hid snd_pcm mei button apple_bl pci_stub vboxpci(O) vboxnetflt(O) vboxnetadp(O) vboxdrv(O) coretemp applesmc input_polldev snd_seq snd_seq_device snd_timer snd soundcore vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost tap uhid hci_vhci
[ 7424.223631]  bluetooth ecdh_generic rfkill ecc crc16 vfio_iommu_type1 vfio uinput userio ppp_generic slhc tun loop nvram cuse fuse dm_crypt sd_mod hid_magicmouse uas usb_storage hid_apple hid_generic usbkbd usbhid hid btrfs libcrc32c crc32c_generic xor amdgpu crct10dif_pclmul crc32_pclmul i915 crc32c_intel ghash_clmulni_intel amd_iommu_v2 hwmon gpu_sched ttm intel_gtt i2c_algo_bit drm_kms_helper ahci raid6_pq aesni_intel xhci_pci syscopyarea libahci sysfillrect sysimgblt fb_sys_fops aes_x86_64 glue_helper xhci_hcd libata sdhci_pci crypto_simd drm cqhci cryptd sdhci usbcore scsi_mod mmc_core agpgart video dm_mirror dm_region_hash dm_log dm_mod
[ 7424.223693] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O      5.3.7_1 #1
[ 7424.223696] Hardware name: Apple Inc. iMac18,3/Mac-BE088AF8C5EB4FA2, BIOS 178.0.0.0.0 09/13/2019
[ 7424.223702] RIP: 0010:dev_watchdog+0x26f/0x280
[ 7424.223708] Code: 37 ac 93 ff eb 88 4c 89 f7 c6 05 9f 1d a4 00 01 e8 26 22 fb ff 44 89 e9 4c 89 f6 48 c7 c7 b8 0b e1 86 48 89 c2 e8 c0 54 9c ff <0f> 0b e9 66 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[ 7424.223712] RSP: 0018:ffff9fb000198e50 EFLAGS: 00010282
[ 7424.223717] RAX: 0000000000000000 RBX: ffff8893a302bd00 RCX: 0000000000000006
[ 7424.223720] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8893aead75d0
[ 7424.223724] RBP: ffff88939ec7045c R08: 0000000000000481 R09: 000000000000004a
[ 7424.223727] R10: ffff9fb000198d08 R11: ffff9fb000198d0d R12: ffff88939ec70480
[ 7424.223729] R13: 0000000000000000 R14: ffff88939ec70000 R15: ffff8893a302bd80
[ 7424.223734] FS:  0000000000000000(0000) GS:ffff8893aeac0000(0000) knlGS:0000000000000000
[ 7424.223737] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7424.223740] CR2: 00007f6d57550000 CR3: 0000000a3d00a004 CR4: 00000000003606e0
[ 7424.223744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7424.223748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7424.223751] Call Trace:
[ 7424.223757]  <IRQ>
[ 7424.223768]  ? qdisc_put_unlocked+0x30/0x30
[ 7424.223774]  ? qdisc_put_unlocked+0x30/0x30
[ 7424.223781]  call_timer_fn+0x32/0x170
[ 7424.223789]  expire_timers+0xab/0x120
[ 7424.223796]  run_timer_softirq+0x8c/0x170
[ 7424.223803]  ? enqueue_hrtimer+0x3d/0xb0
[ 7424.223811]  ? sched_clock+0x5/0x10
[ 7424.223819]  ? sched_clock_cpu+0x11/0xc0
[ 7424.223829]  __do_softirq+0xee/0x341
[ 7424.223840]  irq_exit+0x8d/0xc0
[ 7424.223849]  smp_apic_timer_interrupt+0x80/0x170
[ 7424.223857]  apic_timer_interrupt+0xf/0x20
[ 7424.223864]  </IRQ>
[ 7424.223874] RIP: 0010:cpuidle_enter_state+0xc7/0x490
[ 7424.223881] Code: 31 ff e8 ac 39 a9 ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 a0 03 00 00 31 ff e8 70 09 af ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 88 85 01 00 00 49 63 cc 4d 29 fd 48 8d 04 49 48 c1 e0
[ 7424.223885] RSP: 0018:ffff9fb0000d3e78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 7424.223892] RAX: ffff8893aeaea600 RBX: ffffffff870c2320 RCX: 000000000000001f
[ 7424.223896] RDX: 0000000000000000 RSI: 000000001e79e892 RDI: 0000000000000000
[ 7424.223900] RBP: ffffbfafffce6300 R08: 000006c0963441b2 R09: 0000000000029f10
[ 7424.223903] R10: ffff8893aeae94c4 R11: ffff8893aeae94a4 R12: 0000000000000004
[ 7424.223905] R13: 000006c0963441b2 R14: 0000000000000000 R15: 000006c0962fb2b5
[ 7424.223916]  ? cpuidle_enter_state+0xa4/0x490
[ 7424.223923]  cpuidle_enter+0x29/0x40
[ 7424.223930]  do_idle+0x1cf/0x250
[ 7424.223938]  cpu_startup_entry+0x19/0x20
[ 7424.223944]  start_secondary+0x16b/0x1c0
[ 7424.223950]  secondary_startup_64+0xa4/0xb0
[ 7424.223956] ---[ end trace 348df2f38649c6c3 ]---
[ 7424.223965] tg3 0000:04:00.0 enp4s0f0: transmit timed out, resetting
[ 7424.765435] hrtimer: interrupt took 4233965 ns
[ 7424.972150] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 1.071 msecs
[ 7424.972164] perf: interrupt took too long (9358 > 2500), lowering kernel.perf_event_max_sample_rate to 21000
[ 7427.336791] tg3 0000:04:00.0 enp4s0f0: 0x00000000: 0x168614e4, 0x00100406, 0x02000001, 0x00800040
[ 7427.336794] tg3 0000:04:00.0 enp4s0f0: 0x00000010: 0x9251000c, 0x00000000, 0x9250000c, 0x00000000
[ 7427.336796] tg3 0000:04:00.0 enp4s0f0: 0x00000020: 0x00000000, 0x00000000, 0x00000000, 0x168614e4
[ 7427.336797] tg3 0000:04:00.0 enp4s0f0: 0x00000030: 0xfffff800, 0x00000048, 0x00000000, 0x000001ff
[ 7427.336798] tg3 0000:04:00.0 enp4s0f0: 0x00000040: 0x00000000, 0x03000000, 0xc8035001, 0x16002008
[ 7427.336800] tg3 0000:04:00.0 enp4s0f0: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[ 7427.336801] tg3 0000:04:00.0 enp4s0f0: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x761b000f
[ 7427.336802] tg3 0000:04:00.0 enp4s0f0: 0x00000070: 0x00001090, 0xff892650, 0x00000000, 0x00000000
[ 7427.336804] tg3 0000:04:00.0 enp4s0f0: 0x00000080: 0x168614e4, 0x800000f8, 0x00000000, 0x000003e0
[ 7427.336805] tg3 0000:04:00.0 enp4s0f0: 0x00000090: 0x00000000, 0x0000006f, 0x00000000, 0x000000de
[ 7427.336806] tg3 0000:04:00.0 enp4s0f0: 0x000000a0: 0x8005ac11, 0x00000002, 0x00000122, 0x00020010
[ 7427.336808] tg3 0000:04:00.0 enp4s0f0: 0x000000b0: 0x05908d80, 0x00105400, 0x00475c11, 0x10110142
[ 7427.336809] tg3 0000:04:00.0 enp4s0f0: 0x000000d0: 0x0000081f, 0x00000400, 0x00000000, 0x00010001
[ 7427.336810] tg3 0000:04:00.0 enp4s0f0: 0x000000f0: 0x00000000, 0x00000000, 0x00000000, 0x57766001
[ 7427.336811] tg3 0000:04:00.0 enp4s0f0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
[ 7427.336812] tg3 0000:04:00.0 enp4s0f0: 0x00000110: 0x00002000, 0x00002000, 0x000000a0, 0x00000000
[ 7427.336815] tg3 0000:04:00.0 enp4s0f0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[ 7427.336817] tg3 0000:04:00.0 enp4s0f0: 0x00000140: 0xe8f2cd26, 0x00009810, 0x00000000, 0x00000000
[ 7427.336818] tg3 0000:04:00.0 enp4s0f0: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000001
[ 7427.336820] tg3 0000:04:00.0 enp4s0f0: 0x00000160: 0x1b010002, 0x00000000, 0x00000000, 0x00000000
[ 7427.336821] tg3 0000:04:00.0 enp4s0f0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[ 7427.336824] tg3 0000:04:00.0 enp4s0f0: 0x000001b0: 0x00010018, 0x08460846, 0x00000000, 0x00000000
[ 7427.336827] tg3 0000:04:00.0 enp4s0f0: 0x00000200: 0x00000000, 0x03000000, 0x00000000, 0x5b000000
[ 7427.336829] tg3 0000:04:00.0 enp4s0f0: 0x00000210: 0x00000000, 0xf1000000, 0x00000000, 0xef000000
[ 7427.336831] tg3 0000:04:00.0 enp4s0f0: 0x00000220: 0x00000000, 0xf4000000, 0x00000000, 0x00000000
[ 7427.336832] tg3 0000:04:00.0 enp4s0f0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000000de
[ 7427.336833] tg3 0000:04:00.0 enp4s0f0: 0x00000280: 0x00000000, 0x000003e0, 0x00000000, 0x00000000
[ 7427.336835] tg3 0000:04:00.0 enp4s0f0: 0x00000290: 0x00000000, 0x000003ff, 0x00000000, 0x00000237
[ 7427.336837] tg3 0000:04:00.0 enp4s0f0: 0x00000300: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336839] tg3 0000:04:00.0 enp4s0f0: 0x00000310: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336840] tg3 0000:04:00.0 enp4s0f0: 0x00000320: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336841] tg3 0000:04:00.0 enp4s0f0: 0x00000330: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336843] tg3 0000:04:00.0 enp4s0f0: 0x00000340: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336845] tg3 0000:04:00.0 enp4s0f0: 0x00000350: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336846] tg3 0000:04:00.0 enp4s0f0: 0x00000360: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336847] tg3 0000:04:00.0 enp4s0f0: 0x00000370: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336849] tg3 0000:04:00.0 enp4s0f0: 0x00000380: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336850] tg3 0000:04:00.0 enp4s0f0: 0x00000390: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336852] tg3 0000:04:00.0 enp4s0f0: 0x000003a0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336854] tg3 0000:04:00.0 enp4s0f0: 0x000003b0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336855] tg3 0000:04:00.0 enp4s0f0: 0x000003c0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336857] tg3 0000:04:00.0 enp4s0f0: 0x000003d0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336858] tg3 0000:04:00.0 enp4s0f0: 0x000003e0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336859] tg3 0000:04:00.0 enp4s0f0: 0x000003f0: 0x00000000, 0x0000006f, 0x00000000, 0x0000006f
[ 7427.336861] tg3 0000:04:00.0 enp4s0f0: 0x00000400: 0x00e04808, 0x00400000, 0x00001000, 0x00000880
[ 7427.336863] tg3 0000:04:00.0 enp4s0f0: 0x00000410: 0x00009810, 0xe8f2cd26, 0x00009810, 0xe8f2cd26
[ 7427.336864] tg3 0000:04:00.0 enp4s0f0: 0x00000420: 0x00009810, 0xe8f2cd26, 0x00009810, 0xe8f2cd26
[ 7427.336865] tg3 0000:04:00.0 enp4s0f0: 0x00000430: 0x00000000, 0x00000000, 0x000003cd, 0x000005f2
[ 7427.336866] tg3 0000:04:00.0 enp4s0f0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
[ 7427.336867] tg3 0000:04:00.0 enp4s0f0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000112
[ 7427.336867] tg3 0000:04:00.0 enp4s0f0: 0x00000460: 0x0000000b, 0x00002620, 0x01ff0006, 0x00000000
[ 7427.336868] tg3 0000:04:00.0 enp4s0f0: 0x00000470: 0x80000000, 0x00000000, 0x00000000, 0x50000000
[ 7427.336869] tg3 0000:04:00.0 enp4s0f0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[ 7427.336870] tg3 0000:04:00.0 enp4s0f0: 0x00000500: 0x00000008, 0x00000001, 0x00000000, 0x00000000
[ 7427.336871] tg3 0000:04:00.0 enp4s0f0: 0x00000590: 0x00900110, 0x00000000, 0x00000000, 0x00000000
[ 7427.336873] tg3 0000:04:00.0 enp4s0f0: 0x000005c0: 0xd812f308, 0x000007ba, 0x00000000, 0x00000000
[ 7427.336874] tg3 0000:04:00.0 enp4s0f0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x01921f04
[ 7427.336876] tg3 0000:04:00.0 enp4s0f0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x07008000
[ 7427.336877] tg3 0000:04:00.0 enp4s0f0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[ 7427.336878] tg3 0000:04:00.0 enp4s0f0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 7427.336879] tg3 0000:04:00.0 enp4s0f0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 7427.336880] tg3 0000:04:00.0 enp4s0f0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 7427.336881] tg3 0000:04:00.0 enp4s0f0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 7427.336882] tg3 0000:04:00.0 enp4s0f0: 0x00000670: 0xe30e81c6, 0x5d6b1ae3, 0x8936f3bb, 0xc51fb128
[ 7427.336883] tg3 0000:04:00.0 enp4s0f0: 0x00000680: 0x62ddd599, 0x6c8b2bea, 0xd246f676, 0x79af7d36
[ 7427.336885] tg3 0000:04:00.0 enp4s0f0: 0x00000690: 0xec0b0c70, 0x6dda81d5, 0x00000000, 0x00000000
[ 7427.336886] tg3 0000:04:00.0 enp4s0f0: 0x00000800: 0x00000fc0, 0xffffffff, 0x00000000, 0x00000000
[ 7427.336887] tg3 0000:04:00.0 enp4s0f0: 0x00000810: 0x0000003f, 0xffffffff, 0x00000000, 0x00000000
[ 7427.336888] tg3 0000:04:00.0 enp4s0f0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 7427.336890] tg3 0000:04:00.0 enp4s0f0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 7427.336890] tg3 0000:04:00.0 enp4s0f0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 7427.336892] tg3 0000:04:00.0 enp4s0f0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 7427.336893] tg3 0000:04:00.0 enp4s0f0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[ 7427.336894] tg3 0000:04:00.0 enp4s0f0: 0x00000880: 0x000001c7, 0x001418b0, 0x00000000, 0x00000003
[ 7427.336895] tg3 0000:04:00.0 enp4s0f0: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000
[ 7427.336897] tg3 0000:04:00.0 enp4s0f0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[ 7427.336898] tg3 0000:04:00.0 enp4s0f0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x05bc0000
[ 7427.336899] tg3 0000:04:00.0 enp4s0f0: 0x00000c80: 0x00073452, 0x00000000, 0x00000000, 0x00000000
[ 7427.336900] tg3 0000:04:00.0 enp4s0f0: 0x00000ce0: 0xffcfe60b, 0x00000000, 0x00720014, 0x000c062c
[ 7427.336901] tg3 0000:04:00.0 enp4s0f0: 0x00000cf0: 0x00000000, 0x4000006f, 0x45a80072, 0x00000000
[ 7427.336903] tg3 0000:04:00.0 enp4s0f0: 0x00001000: 0x00000002, 0x00000000, 0xa0007e7f, 0x00000000
[ 7427.336904] tg3 0000:04:00.0 enp4s0f0: 0x00001010: 0x006f06f0, 0x00017e7f, 0x00000000, 0x00000000
[ 7427.336905] tg3 0000:04:00.0 enp4s0f0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000005
[ 7427.336906] tg3 0000:04:00.0 enp4s0f0: 0x00001440: 0x0000006f, 0x0000006f, 0x0000006f, 0x0000006f
[ 7427.336908] tg3 0000:04:00.0 enp4s0f0: 0x00001450: 0x0000006f, 0x0000006f, 0x0000006f, 0x0000006f
[ 7427.336909] tg3 0000:04:00.0 enp4s0f0: 0x00001460: 0x0000006f, 0x0000006f, 0x0000006f, 0x0000006f
[ 7427.336910] tg3 0000:04:00.0 enp4s0f0: 0x00001470: 0x0000006f, 0x0000006f, 0x0000006f, 0x0000006f
[ 7427.336911] tg3 0000:04:00.0 enp4s0f0: 0x00001800: 0x00000016, 0x00000000, 0x0000006f, 0x00000000
[ 7427.336912] tg3 0000:04:00.0 enp4s0f0: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
[ 7427.336913] tg3 0000:04:00.0 enp4s0f0: 0x00001850: 0x0000001f, 0x00000000, 0x000042d0, 0x006f006f
[ 7427.336914] tg3 0000:04:00.0 enp4s0f0: 0x00001860: 0x02000000, 0x00000000, 0x00000000, 0x00000000
[ 7427.336915] tg3 0000:04:00.0 enp4s0f0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 7427.336916] tg3 0000:04:00.0 enp4s0f0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 7427.336918] tg3 0000:04:00.0 enp4s0f0: 0x00002010: 0x00000181, 0x00000001, 0x007bffff, 0x00000000
[ 7427.336919] tg3 0000:04:00.0 enp4s0f0: 0x00002100: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336920] tg3 0000:04:00.0 enp4s0f0: 0x00002110: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336921] tg3 0000:04:00.0 enp4s0f0: 0x00002120: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336922] tg3 0000:04:00.0 enp4s0f0: 0x00002130: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336923] tg3 0000:04:00.0 enp4s0f0: 0x00002140: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336924] tg3 0000:04:00.0 enp4s0f0: 0x00002150: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336925] tg3 0000:04:00.0 enp4s0f0: 0x00002160: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336926] tg3 0000:04:00.0 enp4s0f0: 0x00002170: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336927] tg3 0000:04:00.0 enp4s0f0: 0x00002180: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336928] tg3 0000:04:00.0 enp4s0f0: 0x00002190: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336929] tg3 0000:04:00.0 enp4s0f0: 0x000021a0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336930] tg3 0000:04:00.0 enp4s0f0: 0x000021b0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336931] tg3 0000:04:00.0 enp4s0f0: 0x000021c0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336932] tg3 0000:04:00.0 enp4s0f0: 0x000021d0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336933] tg3 0000:04:00.0 enp4s0f0: 0x000021e0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336934] tg3 0000:04:00.0 enp4s0f0: 0x000021f0: 0x000cea76, 0x000c5235, 0x00000014, 0x00000000
[ 7427.336935] tg3 0000:04:00.0 enp4s0f0: 0x00002200: 0x000d5bc4, 0x00000000, 0x00000000, 0x00000000
[ 7427.336936] tg3 0000:04:00.0 enp4s0f0: 0x00002250: 0x00000004, 0x00000000, 0x00000000, 0x00000000
[ 7427.336938] tg3 0000:04:00.0 enp4s0f0: 0x00002400: 0x00000012, 0x00000000, 0x00000000, 0x00000000
[ 7427.336938] tg3 0000:04:00.0 enp4s0f0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000
[ 7427.336940] tg3 0000:04:00.0 enp4s0f0: 0x00002450: 0x00000000, 0xffff0000, 0x02001800, 0x00006000
[ 7427.336941] tg3 0000:04:00.0 enp4s0f0: 0x00002470: 0x00000000, 0x00000021, 0x00000000, 0x00000000
[ 7427.336942] tg3 0000:04:00.0 enp4s0f0: 0x000024c0: 0x02015101, 0x00000000, 0x00000000, 0x00000000
[ 7427.336943] tg3 0000:04:00.0 enp4s0f0: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[ 7427.336944] tg3 0000:04:00.0 enp4s0f0: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x00000080
[ 7427.336945] tg3 0000:04:00.0 enp4s0f0: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[ 7427.336947] tg3 0000:04:00.0 enp4s0f0: 0x00002c20: 0x00000004, 0x00000000, 0x00000000, 0x00000000
[ 7427.336948] tg3 0000:04:00.0 enp4s0f0: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x00000080
[ 7427.336949] tg3 0000:04:00.0 enp4s0f0: 0x00003600: 0x00137400, 0x00130000, 0x00110000, 0x00000000
[ 7427.336950] tg3 0000:04:00.0 enp4s0f0: 0x00003610: 0x00170000, 0x00130000, 0x00150000, 0x00000000
[ 7427.336951] tg3 0000:04:00.0 enp4s0f0: 0x00003620: 0x00000000, 0x00000000, 0x00000000, 0x00022080
[ 7427.336952] tg3 0000:04:00.0 enp4s0f0: 0x00003630: 0x00000000, 0x80008000, 0x00001080, 0x00000000
[ 7427.336954] tg3 0000:04:00.0 enp4s0f0: 0x00003640: 0x0000000a, 0x00400000, 0x00000020, 0x00000019
[ 7427.336955] tg3 0000:04:00.0 enp4s0f0: 0x00003650: 0x00000171, 0x00000bff, 0x57766001, 0x00000000
[ 7427.336956] tg3 0000:04:00.0 enp4s0f0: 0x00003660: 0x00000000, 0x00000000, 0x010400a7, 0x00000000
[ 7427.336957] tg3 0000:04:00.0 enp4s0f0: 0x00003670: 0x0000002a, 0xfeff7fe3, 0x0000000a, 0x00000000
[ 7427.336958] tg3 0000:04:00.0 enp4s0f0: 0x00003690: 0x80157766, 0x00000000, 0x00000000, 0x00000000
[ 7427.336959] tg3 0000:04:00.0 enp4s0f0: 0x000036a0: 0x00000000, 0x00000000, 0x00000000, 0x00000044
[ 7427.336960] tg3 0000:04:00.0 enp4s0f0: 0x000036b0: 0x00100348, 0x07ff07ff, 0x07ff07ff, 0x0100000d
[ 7427.336961] tg3 0000:04:00.0 enp4s0f0: 0x000036c0: 0xfffffffe, 0x00000000, 0x00000000, 0x00000128
[ 7427.336962] tg3 0000:04:00.0 enp4s0f0: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x000012ba
[ 7427.336963] tg3 0000:04:00.0 enp4s0f0: 0x000036e0: 0x00000b19, 0x20800799, 0x00500799, 0x00000000
[ 7427.336964] tg3 0000:04:00.0 enp4s0f0: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048
[ 7427.336965] tg3 0000:04:00.0 enp4s0f0: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
[ 7427.336966] tg3 0000:04:00.0 enp4s0f0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[ 7427.336967] tg3 0000:04:00.0 enp4s0f0: 0x00003c30: 0x00000000, 0x00000000, 0x00000000, 0xffffa000
[ 7427.336968] tg3 0000:04:00.0 enp4s0f0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[ 7427.336969] tg3 0000:04:00.0 enp4s0f0: 0x00003c50: 0x00000000, 0x00000017, 0x00000000, 0x00000000
[ 7427.336970] tg3 0000:04:00.0 enp4s0f0: 0x00003c80: 0x000003e0, 0x00000001, 0x000003ff, 0x00000237
[ 7427.336972] tg3 0000:04:00.0 enp4s0f0: 0x00003cc0: 0x0000006f, 0x00000000, 0x00000000, 0x00000000
[ 7427.336972] tg3 0000:04:00.0 enp4s0f0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[ 7427.336974] tg3 0000:04:00.0 enp4s0f0: 0x00003d00: 0x00000000, 0xffff9000, 0x00000000, 0xffff8000
[ 7427.336975] tg3 0000:04:00.0 enp4s0f0: 0x00003d10: 0x00000000, 0xffff7000, 0x00000000, 0xffff6000
[ 7427.336976] tg3 0000:04:00.0 enp4s0f0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 7427.336977] tg3 0000:04:00.0 enp4s0f0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 7427.336978] tg3 0000:04:00.0 enp4s0f0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 7427.336979] tg3 0000:04:00.0 enp4s0f0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 7427.336980] tg3 0000:04:00.0 enp4s0f0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 7427.336981] tg3 0000:04:00.0 enp4s0f0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 7427.336982] tg3 0000:04:00.0 enp4s0f0: 0x00004000: 0x00000002, 0x00000000, 0x00069949, 0x00012848
[ 7427.336983] tg3 0000:04:00.0 enp4s0f0: 0x00004010: 0x0047f020, 0x00235012, 0x008c5830, 0x01005e42
[ 7427.336984] tg3 0000:04:00.0 enp4s0f0: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050
[ 7427.336985] tg3 0000:04:00.0 enp4s0f0: 0x00004030: 0x00000000, 0x01085c20, 0x00258010, 0x0047f002
[ 7427.336986] tg3 0000:04:00.0 enp4s0f0: 0x00004040: 0x00480000, 0x009b0600, 0x00000010, 0x0047e062
[ 7427.336987] tg3 0000:04:00.0 enp4s0f0: 0x00004400: 0x00000006, 0x00000000, 0x00010000, 0x0000a000
[ 7427.336988] tg3 0000:04:00.0 enp4s0f0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[ 7427.336990] tg3 0000:04:00.0 enp4s0f0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[ 7427.336992] tg3 0000:04:00.0 enp4s0f0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x07e1fc81
[ 7427.336994] tg3 0000:04:00.0 enp4s0f0: 0x00004450: 0x023c0422, 0x00590037, 0x00000000, 0x00000000
[ 7427.336996] tg3 0000:04:00.0 enp4s0f0: 0x00004700: 0x00030002, 0x00200000, 0x00200010, 0x9fa90000
[ 7427.336999] tg3 0000:04:00.0 enp4s0f0: 0x00004710: 0xffff46a0, 0x000c5914, 0x00100011, 0x00000000
[ 7427.337000] tg3 0000:04:00.0 enp4s0f0: 0x00004720: 0x00000000, 0x00000000, 0xf02c0000, 0xffff46f0
[ 7427.337002] tg3 0000:04:00.0 enp4s0f0: 0x00004770: 0x000c0404, 0x00000042, 0x00000018, 0x00000000
[ 7427.337003] tg3 0000:04:00.0 enp4s0f0: 0x00004800: 0x188303fe, 0x00000000, 0x00000000, 0x00000020
[ 7427.337004] tg3 0000:04:00.0 enp4s0f0: 0x00004810: 0x00000000, 0x0000000a, 0x00429574, 0x00001514
[ 7427.337005] tg3 0000:04:00.0 enp4s0f0: 0x00004820: 0x00000080, 0x00000000, 0xf0330000, 0xffcfe620
[ 7427.337007] tg3 0000:04:00.0 enp4s0f0: 0x00004840: 0x00000230, 0x00000230, 0x000e2200, 0x8042f831
[ 7427.337008] tg3 0000:04:00.0 enp4s0f0: 0x00004850: 0xbf973007, 0xce280056, 0x7f7e7f7e, 0x00000000
[ 7427.337009] tg3 0000:04:00.0 enp4s0f0: 0x00004860: 0x00000080, 0x00000000, 0x00000000, 0x00000000
[ 7427.337010] tg3 0000:04:00.0 enp4s0f0: 0x00004870: 0x00000000, 0x003e1820, 0x003e1820, 0x00000000
[ 7427.337012] tg3 0000:04:00.0 enp4s0f0: 0x00004890: 0x28090404, 0x00305400, 0x00000000, 0x00000000
[ 7427.337013] tg3 0000:04:00.0 enp4s0f0: 0x000048a0: 0x000f0010, 0x00000000, 0x00000000, 0x00000000
[ 7427.337014] tg3 0000:04:00.0 enp4s0f0: 0x00004900: 0x18030006, 0x00000003, 0x30000000, 0x00000020
[ 7427.337016] tg3 0000:04:00.0 enp4s0f0: 0x00004910: 0x00000040, 0x00000003, 0x0000d570, 0x00000000
[ 7427.337017] tg3 0000:04:00.0 enp4s0f0: 0x00004920: 0x00000071, 0x02378000, 0xff9f0401, 0x0f0000a0
[ 7427.337018] tg3 0000:04:00.0 enp4s0f0: 0x00004930: 0xffe24601, 0x0f0000a0, 0xffcc1ce9, 0x0f000048
[ 7427.337019] tg3 0000:04:00.0 enp4s0f0: 0x00004940: 0xffcd1001, 0x0f000068, 0x5252a0a0, 0xbcbc1515
[ 7427.337021] tg3 0000:04:00.0 enp4s0f0: 0x00004950: 0xf0330000, 0xff9f04a0, 0xaf000000, 0x716f0070
[ 7427.337022] tg3 0000:04:00.0 enp4s0f0: 0x00004960: 0x00000000, 0xffe246a0, 0xaf000000, 0x4000006d
[ 7427.337023] tg3 0000:04:00.0 enp4s0f0: 0x00004970: 0x00028203, 0x00205400, 0x0000001c, 0x000000ff
[ 7427.337024] tg3 0000:04:00.0 enp4s0f0: 0x00004980: 0x00000071, 0x00000000, 0x00000000, 0x00000000
[ 7427.337025] tg3 0000:04:00.0 enp4s0f0: 0x00004990: 0x00000000, 0x00000000, 0xffcc1d30, 0xef000000
[ 7427.337026] tg3 0000:04:00.0 enp4s0f0: 0x000049a0: 0x00f00024, 0x00000071, 0x00000000, 0x00000000
[ 7427.337027] tg3 0000:04:00.0 enp4s0f0: 0x000049b0: 0xff549801, 0xff721a01, 0xffdd3401, 0xff6ef201
[ 7427.337028] tg3 0000:04:00.0 enp4s0f0: 0x000049c0: 0xff9f0402, 0xffe24602, 0xffcc1cee, 0xffcd1000
[ 7427.337029] tg3 0000:04:00.0 enp4s0f0: 0x000049d0: 0xff549802, 0xff721a02, 0xffdd3402, 0xff6ef202
[ 7427.337030] tg3 0000:04:00.0 enp4s0f0: 0x000049f0: 0xf0310000, 0xffcd1068, 0x8f000000, 0x00007fbf
[ 7427.337031] tg3 0000:04:00.0 enp4s0f0: 0x00004c00: 0x200003fe, 0x00000020, 0x00000000, 0x0000000a
[ 7427.337033] tg3 0000:04:00.0 enp4s0f0: 0x00004c10: 0x00000030, 0x00000001, 0x00000006, 0x00000000
[ 7427.337034] tg3 0000:04:00.0 enp4s0f0: 0x00004c20: 0x00000000, 0x00000020, 0x00000000, 0x00000006
[ 7427.337035] tg3 0000:04:00.0 enp4s0f0: 0x00004c30: 0x00000000, 0x00f80000, 0x000000cf, 0x000000cf
[ 7427.337037] tg3 0000:04:00.0 enp4s0f0: 0x00004c40: 0x80040000, 0xfffc8000, 0x00010020, 0x00120020
[ 7427.337038] tg3 0000:04:00.0 enp4s0f0: 0x00004c50: 0x3fe003df, 0x016ffe36, 0x637ffc00, 0x55170303
[ 7427.337039] tg3 0000:04:00.0 enp4s0f0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[ 7427.337040] tg3 0000:04:00.0 enp4s0f0: 0x00005000: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 7427.337041] tg3 0000:04:00.0 enp4s0f0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08000898
[ 7427.337042] tg3 0000:04:00.0 enp4s0f0: 0x00005020: 0x03e00008, 0x00000000, 0x00000000, 0x40000020
[ 7427.337043] tg3 0000:04:00.0 enp4s0f0: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337044] tg3 0000:04:00.0 enp4s0f0: 0x00005040: 0x00000000, 0x00000000, 0x080008be, 0x00000000
[ 7427.337045] tg3 0000:04:00.0 enp4s0f0: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 7427.337046] tg3 0000:04:00.0 enp4s0f0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x0800094c
[ 7427.337047] tg3 0000:04:00.0 enp4s0f0: 0x000050a0: 0xafa20014, 0x00000000, 0x00000000, 0x40000020
[ 7427.337048] tg3 0000:04:00.0 enp4s0f0: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337049] tg3 0000:04:00.0 enp4s0f0: 0x000050c0: 0x00000000, 0x00000000, 0x08000cc8, 0x00000000
[ 7427.337051] tg3 0000:04:00.0 enp4s0f0: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 7427.337052] tg3 0000:04:00.0 enp4s0f0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08000878
[ 7427.337053] tg3 0000:04:00.0 enp4s0f0: 0x00005120: 0x8f4301e8, 0x00000000, 0x00000000, 0x40000020
[ 7427.337054] tg3 0000:04:00.0 enp4s0f0: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337055] tg3 0000:04:00.0 enp4s0f0: 0x00005140: 0x00000000, 0x00000000, 0x08000cc8, 0x00000000
[ 7427.337056] tg3 0000:04:00.0 enp4s0f0: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 7427.337058] tg3 0000:04:00.0 enp4s0f0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000cc8
[ 7427.337059] tg3 0000:04:00.0 enp4s0f0: 0x000051a0: 0x30420080, 0x00000000, 0x00000000, 0x40000020
[ 7427.337060] tg3 0000:04:00.0 enp4s0f0: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337061] tg3 0000:04:00.0 enp4s0f0: 0x000051c0: 0x00000000, 0x00000000, 0x08000a80, 0x00000000
[ 7427.337062] tg3 0000:04:00.0 enp4s0f0: 0x00005200: 0x0800ff70, 0x10400004, 0x21122010, 0x10400004
[ 7427.337063] tg3 0000:04:00.0 enp4s0f0: 0x00005210: 0x08000cc0, 0x10400004, 0x0800ff70, 0x10400004
[ 7427.337064] tg3 0000:04:00.0 enp4s0f0: 0x00005220: 0x0800ff70, 0x10400004, 0x21122010, 0x10400004
[ 7427.337065] tg3 0000:04:00.0 enp4s0f0: 0x00005230: 0x00000000, 0x10400004, 0x00000000, 0x10400004
[ 7427.337066] tg3 0000:04:00.0 enp4s0f0: 0x00005240: 0x00000000, 0x10400004, 0x00000000, 0xc0000000
[ 7427.337067] tg3 0000:04:00.0 enp4s0f0: 0x00005250: 0x00000000, 0x00000001, 0x00000000, 0xc0000000
[ 7427.337068] tg3 0000:04:00.0 enp4s0f0: 0x00005260: 0x00000000, 0xc0000000, 0x0800ff98, 0x0800ff70
[ 7427.337069] tg3 0000:04:00.0 enp4s0f0: 0x00005270: 0x0800ff98, 0x0800ff70, 0xc0000000, 0x0100000d
[ 7427.337070] tg3 0000:04:00.0 enp4s0f0: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 7427.337072] tg3 0000:04:00.0 enp4s0f0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000878
[ 7427.337073] tg3 0000:04:00.0 enp4s0f0: 0x000052a0: 0x30420001, 0x00000000, 0x00000000, 0x40000020
[ 7427.337074] tg3 0000:04:00.0 enp4s0f0: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337075] tg3 0000:04:00.0 enp4s0f0: 0x000052c0: 0x00000000, 0x00000000, 0x0800095c, 0x00000000
[ 7427.337076] tg3 0000:04:00.0 enp4s0f0: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 7427.337077] tg3 0000:04:00.0 enp4s0f0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08000bf0
[ 7427.337078] tg3 0000:04:00.0 enp4s0f0: 0x00005320: 0xafbf0024, 0x00000000, 0x00000000, 0x40000020
[ 7427.337079] tg3 0000:04:00.0 enp4s0f0: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337080] tg3 0000:04:00.0 enp4s0f0: 0x00005340: 0x00000000, 0x00000000, 0x080009ea, 0x00000000
[ 7427.337081] tg3 0000:04:00.0 enp4s0f0: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 7427.337083] tg3 0000:04:00.0 enp4s0f0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08000890
[ 7427.337084] tg3 0000:04:00.0 enp4s0f0: 0x000053a0: 0x14600031, 0x00000000, 0x00000000, 0x40000020
[ 7427.337085] tg3 0000:04:00.0 enp4s0f0: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 7427.337086] tg3 0000:04:00.0 enp4s0f0: 0x000053c0: 0x00000000, 0x00000000, 0x0800095c, 0x00000000
[ 7427.337087] tg3 0000:04:00.0 enp4s0f0: 0x00005800: 0x00000000, 0x03000000, 0x00000000, 0x5b000000
[ 7427.337088] tg3 0000:04:00.0 enp4s0f0: 0x00005810: 0x00000000, 0xf1000000, 0x00000000, 0xef000000
[ 7427.337089] tg3 0000:04:00.0 enp4s0f0: 0x00005820: 0x00000000, 0xf4000000, 0x00000000, 0x00000000
[ 7427.337090] tg3 0000:04:00.0 enp4s0f0: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x000000de
[ 7427.337091] tg3 0000:04:00.0 enp4s0f0: 0x00005880: 0x00000000, 0x000003e0, 0x00000000, 0x00000000
[ 7427.337092] tg3 0000:04:00.0 enp4s0f0: 0x00005890: 0x00000000, 0x000003ff, 0x00000000, 0x00000237
[ 7427.337095] tg3 0000:04:00.0 enp4s0f0: 0x00005900: 0x00000000, 0x0000006f, 0x00000000, 0x00000000
[ 7427.337096] tg3 0000:04:00.0 enp4s0f0: 0x00005980: 0x0000006f, 0x00000000, 0x00000000, 0x00000000
[ 7427.337098] tg3 0000:04:00.0 enp4s0f0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[ 7427.337100] tg3 0000:04:00.0 enp4s0f0: 0x00006000: 0x20000082, 0x00000000, 0x00000000, 0x00000000
[ 7427.337102] tg3 0000:04:00.0 enp4s0f0: 0x00006400: 0x00000000, 0x00000000, 0x00011091, 0xc0000000
[ 7427.337103] tg3 0000:04:00.0 enp4s0f0: 0x00006410: 0x05000016, 0x05000016, 0x00000000, 0x00000000
[ 7427.337104] tg3 0000:04:00.0 enp4s0f0: 0x00006430: 0x00000000, 0x14e41686, 0x168614e4, 0x01020000
[ 7427.337105] tg3 0000:04:00.0 enp4s0f0: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[ 7427.337106] tg3 0000:04:00.0 enp4s0f0: 0x000064c0: 0x00000005, 0x00000002, 0x00000122, 0x00000000
[ 7427.337107] tg3 0000:04:00.0 enp4s0f0: 0x000064d0: 0x00000040, 0x00008d80, 0x00000000, 0x00d75e11
[ 7427.337108] tg3 0000:04:00.0 enp4s0f0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[ 7427.337109] tg3 0000:04:00.0 enp4s0f0: 0x000064f0: 0x00000002, 0x00000030, 0x00000000, 0x00000000
[ 7427.337110] tg3 0000:04:00.0 enp4s0f0: 0x00006500: 0x03e10003, 0xe8f2cd26, 0x00009810, 0x00000003
[ 7427.337111] tg3 0000:04:00.0 enp4s0f0: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000
[ 7427.337112] tg3 0000:04:00.0 enp4s0f0: 0x00006550: 0x00000001, 0x02800000, 0x0000000f, 0x00000000
[ 7427.337113] tg3 0000:04:00.0 enp4s0f0: 0x00006560: 0x0000000f, 0x00000000, 0x00000000, 0x00000000
[ 7427.337113] tg3 0000:04:00.0 enp4s0f0: 0x00006800: 0x04130034, 0x20081082, 0x01009618, 0xb8854ad8
[ 7427.337114] tg3 0000:04:00.0 enp4s0f0: 0x00006810: 0x21122010, 0xffffffff, 0x00000000, 0x00000040
[ 7427.337115] tg3 0000:04:00.0 enp4s0f0: 0x00006880: 0x77fff020, 0x00000040, 0x80011686, 0x00000000
[ 7427.337116] tg3 0000:04:00.0 enp4s0f0: 0x00006890: 0x00800000, 0x88008000, 0x00000000, 0x00000000
[ 7427.337116] tg3 0000:04:00.0 enp4s0f0: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000
[ 7427.337117] tg3 0000:04:00.0 enp4s0f0: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000
[ 7427.337118] tg3 0000:04:00.0 enp4s0f0: 0x00006900: 0x5dd32208, 0x00000000, 0x00000000, 0x00000000
[ 7427.337119] tg3 0000:04:00.0 enp4s0f0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[ 7427.337120] tg3 0000:04:00.0 enp4s0f0: 0x00007000: 0x08000188, 0x00000000, 0x00000000, 0x000000c4
[ 7427.337120] tg3 0000:04:00.0 enp4s0f0: 0x00007010: 0xd1c00001, 0x50408272, 0x00d70081, 0x03008200
[ 7427.337121] tg3 0000:04:00.0 enp4s0f0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[ 7427.337122] tg3 0000:04:00.0 enp4s0f0: 0x00007030: 0x0001f200, 0x000000c8, 0x00030030, 0x00000000
[ 7427.337123] tg3 0000:04:00.0 enp4s0f0: 0x00007500: 0x00000000, 0x00000000, 0x00000080, 0x00000000
[ 7427.337125] tg3 0000:04:00.0 enp4s0f0: 0: Host status block [00000001:00000003:(0000:0010:0000):(0000:004e)]
[ 7427.337126] tg3 0000:04:00.0 enp4s0f0: 0: NAPI info [00000003:00000003:(006f:004e:01ff):0000:(00de:0000:0000:0000)]
[ 7427.337127] tg3 0000:04:00.0 enp4s0f0: 1: Host status block [00000001:0000005b:(0000:0000:0000):(03e0:0000)]
[ 7427.337128] tg3 0000:04:00.0 enp4s0f0: 1: NAPI info [0000005b:0000005b:(0000:0000:01ff):03e0:(01e0:01e0:0000:0000)]
[ 7427.337129] tg3 0000:04:00.0 enp4s0f0: 2: Host status block [00000001:000000f1:(0000:0000:0000):(0000:0000)]
[ 7427.337130] tg3 0000:04:00.0 enp4s0f0: 2: NAPI info [000000f1:000000f1:(0000:0000:01ff):0000:(0000:0000:0000:0000)]
[ 7427.337131] tg3 0000:04:00.0 enp4s0f0: 3: Host status block [00000001:000000ef:(0000:0000:0000):(0000:0000)]
[ 7427.337132] tg3 0000:04:00.0 enp4s0f0: 3: NAPI info [000000ef:000000ef:(0000:0000:01ff):03ff:(01ff:01ff:0000:0000)]
[ 7427.337133] tg3 0000:04:00.0 enp4s0f0: 4: Host status block [00000001:000000f4:(0000:0000:0237):(0000:0000)]
[ 7427.337134] tg3 0000:04:00.0 enp4s0f0: 4: NAPI info [000000f4:000000f4:(0000:0000:01ff):0237:(0037:0037:0000:0000)]
[ 7427.471786] tg3 0000:04:00.0: tg3_stop_block timed out, ofs=1400 enable_bit=2
[ 7427.606241] tg3 0000:04:00.0: tg3_stop_block timed out, ofs=4c00 enable_bit=2
[ 7427.632603] tg3 0000:04:00.0 enp4s0f0: Link is down
[ 7431.326885] tg3 0000:04:00.0 enp4s0f0: Link is up at 1000 Mbps, full duplex
[ 7431.326901] tg3 0000:04:00.0 enp4s0f0: Flow control is on for TX and on for RX
[ 7431.326905] tg3 0000:04:00.0 enp4s0f0: EEE is disabled

Comment 32 Nathan Shearer 2019-11-23 06:53:44 UTC

I can easily reproduce this on a Mac Mini 6.1 with vanilla sources 5.3.11 by simply using the computer. Any excessive network usage for extended periods will cause the ethernet to hang and corrupts all kinds of network activity as a result.

Disabling tso and gso with ethtool seems to temporarily fix the crash.
https://bugs.launchpad.net/dell-poweredge/+bug/1331513

ethtool -K eth0 tso off
ethtool -K eth0 gso off

Nothing special is needed to trigger it. Just send/receive lots of data (maybe around 100GB).

# lspci -v
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at a0400000 (64-bit, prefetchable) [size=64K]
        Memory at a0410000 (64-bit, prefetchable) [size=64K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number 00-00-10-dd-b1-e4-72-3f
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [160] Virtual Channel
        Capabilities: [1b0] Latency Tolerance Reporting
        Kernel driver in use: tg3
        Kernel modules: tg3

Nov 22 20:08:16 iscsi03 kernel: ------------[ cut here ]------------
Nov 22 20:08:16 iscsi03 kernel: NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Nov 22 20:08:16 iscsi03 kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x265/0x270
Nov 22 20:08:16 iscsi03 kernel: Modules linked in: intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi kvm_intel binfmt_misc kvm snd_hda_codec_cirrus b43 snd_hda_codec_generic irqbypass cordic ledtrig_audio mac80211 iTCO_wdt crct10dif_pclmul mousedev crc32_pclmul crc32c_intel hid_appleir btusb sdhci_pci cfg80211 ghash_clmulni_intel btrtl hid_generic aesni_intel cqhci aes_x86_64 btbcm i915 btintel applesmc crypto_simd firewire_ohci cryptd sdhci firewire_core tg3 snd_hda_intel glue_helper snd_hda_codec cec mei_me apple_gmux snd_hda_core apple_bl lpc_ich mei snd_hwdep pcspkr video efivarfs
Nov 22 20:08:16 iscsi03 kernel: CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.3.11 #1
Nov 22 20:08:16 iscsi03 kernel: Hardware name: Apple Inc. Macmini6,1/Mac-031AEE4D24BFF0B1, BIOS MM61.88Z.010E.B00.1804111136 04/11/2018
Nov 22 20:08:16 iscsi03 kernel: RIP: 0010:dev_watchdog+0x265/0x270
Nov 22 20:08:16 iscsi03 kernel: Code: b1 f7 44 ff eb 88 4c 89 f7 c6 05 76 7d fb 00 01 e8 b0 65 fb ff 44 89 e9 4c 89 f6 48 c7 c7 f0 e8 ba 9d 48 89 c2 e8 94 b4 4c ff <0f> 0b e9 66 ff ff ff 0f 1f 40 00 48 c7 47 08 00 00 00 00 48 c7 07
Nov 22 20:08:16 iscsi03 kernel: RSP: 0018:ffffa9c280138e68 EFLAGS: 00010286
Nov 22 20:08:16 iscsi03 kernel: RAX: 0000000000000000 RBX: ffff8a8a265acd00 RCX: 0000000000000000
Nov 22 20:08:16 iscsi03 kernel: RDX: 0000000000000103 RSI: 0000000000000086 RDI: 00000000ffffffff
Nov 22 20:08:16 iscsi03 kernel: RBP: ffff8a8a2837c45c R08: 0000000000000466 R09: 0000000000000001
Nov 22 20:08:16 iscsi03 kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff8a8a2837c480
Nov 22 20:08:16 iscsi03 kernel: R13: 0000000000000000 R14: ffff8a8a2837c000 R15: ffff8a8a265acd80
Nov 22 20:08:16 iscsi03 kernel: FS:  0000000000000000(0000) GS:ffff8a8a2b180000(0000) knlGS:0000000000000000
Nov 22 20:08:16 iscsi03 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 22 20:08:16 iscsi03 kernel: CR2: 00007f656000a248 CR3: 000000003060a001 CR4: 00000000001606e0
Nov 22 20:08:16 iscsi03 kernel: Call Trace:
Nov 22 20:08:16 iscsi03 kernel:  <IRQ>
Nov 22 20:08:16 iscsi03 kernel:  ? qdisc_put_unlocked+0x30/0x30
Nov 22 20:08:16 iscsi03 kernel:  ? qdisc_put_unlocked+0x30/0x30
Nov 22 20:08:16 iscsi03 kernel:  call_timer_fn+0x28/0x150
Nov 22 20:08:16 iscsi03 kernel:  run_timer_softirq+0x188/0x4c0
Nov 22 20:08:16 iscsi03 kernel:  ? enqueue_hrtimer+0x31/0xa0
Nov 22 20:08:16 iscsi03 kernel:  ? __hrtimer_run_queues+0x120/0x2c0
Nov 22 20:08:16 iscsi03 kernel:  __do_softirq+0x104/0x332
Nov 22 20:08:16 iscsi03 kernel:  irq_exit+0x99/0xb0
Nov 22 20:08:16 iscsi03 kernel:  smp_apic_timer_interrupt+0xa1/0x1a0
Nov 22 20:08:16 iscsi03 kernel:  apic_timer_interrupt+0xf/0x20
Nov 22 20:08:16 iscsi03 kernel:  </IRQ>
Nov 22 20:08:16 iscsi03 kernel: RIP: 0010:cpuidle_enter_state+0xbf/0x450
Nov 22 20:08:16 iscsi03 kernel: Code: e8 26 76 68 ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 6b 03 00 00 31 ff e8 58 5e 6e ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 89 01 02 00 00 c7 45 10 00 00 00 00 48 83 c4 18 44 89
Nov 22 20:08:16 iscsi03 kernel: RSP: 0018:ffffa9c2800abe68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Nov 22 20:08:16 iscsi03 kernel: RAX: ffff8a8a2b180000 RBX: ffffffff9dec03c0 RCX: 000000000000001f
Nov 22 20:08:16 iscsi03 kernel: RDX: 0000000000000000 RSI: 000000003350f535 RDI: 0000000000000000
Nov 22 20:08:16 iscsi03 kernel: RBP: ffffc9c27fd9e100 R08: 0000012b7fcaa57c R09: 000000000000e8a1
Nov 22 20:08:16 iscsi03 kernel: R10: ffff8a8a2b1a9184 R11: ffff8a8a2b1a9164 R12: 0000000000000005
Nov 22 20:08:16 iscsi03 kernel: R13: 0000012b7fcaa57c R14: 0000000000000005 R15: ffff8a8a2a574740
Nov 22 20:08:16 iscsi03 kernel:  ? cpuidle_enter_state+0x9a/0x450
Nov 22 20:08:16 iscsi03 kernel:  cpuidle_enter+0x24/0x40
Nov 22 20:08:16 iscsi03 kernel:  do_idle+0x1d4/0x250
Nov 22 20:08:16 iscsi03 kernel:  cpu_startup_entry+0x14/0x20
Nov 22 20:08:16 iscsi03 kernel:  start_secondary+0x159/0x190
Nov 22 20:08:16 iscsi03 kernel:  secondary_startup_64+0xa4/0xb0
Nov 22 20:08:16 iscsi03 kernel: ---[ end trace 3611b8dfb58a5a06 ]---
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: transmit timed out, resetting
Nov 22 20:08:16 iscsi03 kernel: hrtimer: interrupt took 3285014 ns
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000000: 0x168614e4, 0x00100406, 0x02000001, 0x00800040
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000010: 0xa040000c, 0x00000000, 0xa041000c, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000020: 0x00000000, 0x00000000, 0x00000000, 0x168614e4
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000100
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000040: 0x00000000, 0x56000000, 0xc8035001, 0x16002008
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x01f802d1
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000070: 0x00001090, 0x64000190, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000080: 0x168614e4, 0x40000040, 0x00000000, 0x000003ff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000090: 0x00000000, 0x000000e8, 0x00000000, 0x0000013f
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000000a0: 0x8005ac11, 0x00000002, 0x00000122, 0x00020010
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000000b0: 0x05908d80, 0x00105400, 0x00475c11, 0x10110142
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000000d0: 0x0000081f, 0x00000000, 0x00000000, 0x00010001
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000000f0: 0x00000000, 0x00000000, 0x00000000, 0x57766001
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000110: 0x00002000, 0x00002000, 0x000000a0, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000140: 0xb1e4723f, 0x000010dd, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000001
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000160: 0x1b010002, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000001b0: 0x00010018, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000200: 0x00000000, 0x56000000, 0x00000000, 0xeb000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000210: 0x00000000, 0x9e000000, 0x00000000, 0xdb000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000220: 0x00000000, 0x99000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x0000013f
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000280: 0x00000000, 0x000003ff, 0x00000000, 0x000001b2
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000290: 0x00000000, 0x00000352, 0x00000000, 0x00000374
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000300: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000310: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000320: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000330: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000340: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000350: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000360: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000370: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000380: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000390: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003a0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003b0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003c0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003d0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003e0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000003f0: 0x00000000, 0x000000e8, 0x00000000, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000400: 0x00e04808, 0x00400000, 0x00001000, 0x00000880
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000410: 0x000010dd, 0xb1e4723f, 0x000010dd, 0xb1e4723f
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000420: 0x000010dd, 0xb1e4723f, 0x000010dd, 0xb1e4723f
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000430: 0x00000000, 0x00000000, 0x00000239, 0x000005f2
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000460: 0x00000008, 0x00002620, 0x01ff0002, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000470: 0x80000000, 0x00000000, 0x00002000, 0x40000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000500: 0x00000008, 0x00000001, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000590: 0x00900110, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000005c0: 0x085603d0, 0x07232206, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x051dc000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000670: 0x13082ec3, 0xdf74eb2e, 0x820598ba, 0x564fee9a
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000680: 0x92dc73ba, 0x5d980d15, 0x40783008, 0xa051fda1
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000690: 0x2278c44b, 0x404ab0c6, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000880: 0x00000000, 0x000171c2, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x002a0000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000c80: 0x00004f9f, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000ce0: 0xffe65002, 0x00000000, 0x0000002a, 0x00040028
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00000cf0: 0x00000000, 0x50000068, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001000: 0x00000002, 0x00000000, 0xa0000303, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001010: 0x00e80e81, 0x00000303, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000001
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001440: 0x000000e8, 0x000000e8, 0x000000e8, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001450: 0x000000e8, 0x000000e8, 0x000000e8, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001460: 0x000000e8, 0x000000e8, 0x000000e8, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001470: 0x000000e8, 0x000000e8, 0x000000e8, 0x000000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001800: 0x00000016, 0x00000000, 0x000000e8, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001850: 0x0000001f, 0x00000000, 0x00004270, 0x00e800e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001860: 0x01000000, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002010: 0x00000181, 0x00000001, 0x007bffff, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002100: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002110: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002120: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002130: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002140: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002150: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002160: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002170: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002180: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002190: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021a0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021b0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021c0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021d0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021e0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000021f0: 0x000e4322, 0x000c120a, 0x00000024, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002200: 0x0000ee1e, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002400: 0x00000012, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002450: 0x00000000, 0xffff0000, 0x02001800, 0x00006000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002470: 0x00000000, 0x00000082, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000024c0: 0x02000101, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x000000e4
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00002c20: 0x00020004, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x000000e4
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003600: 0x00137400, 0x00130000, 0x00110000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003610: 0x00170000, 0x00130000, 0x00150000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003620: 0x00000000, 0x00000000, 0x00000000, 0x00022080
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003630: 0x00000000, 0x80008000, 0x00001080, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003640: 0x0000000a, 0x00400000, 0x00000020, 0x00000019
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003650: 0x00000171, 0x00000bff, 0x57766001, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003660: 0x00000000, 0x00000000, 0x010400a7, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003670: 0x0000002a, 0xfeffffe5, 0x0000000a, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003690: 0x80157766, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000036a0: 0x00000000, 0x00000000, 0x00000000, 0x00000044
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000036b0: 0x00100348, 0x07ff07ff, 0x07ff07ff, 0x0100000d
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000036c0: 0xfffffffe, 0x00000000, 0x00000000, 0x00000029
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x000012af
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000036e0: 0x00000b19, 0x20800799, 0x00500799, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c30: 0x00000000, 0x00000000, 0x00000000, 0xffffb000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c50: 0x00000000, 0x00000078, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003c80: 0x00000000, 0x000001b2, 0x00000352, 0x00000374
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003cc0: 0x000000e8, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003d00: 0x00000000, 0xffffa000, 0x00000000, 0xffff9000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003d10: 0x00000000, 0xffff8000, 0x00000000, 0xffff7000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004000: 0x00000002, 0x00000000, 0x00143323, 0x00010880
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004010: 0x00403660, 0x0020b012, 0x00800440, 0x01005022
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004030: 0x00000000, 0x01084e20, 0x00310010, 0x00487002
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004040: 0x00489000, 0x02026600, 0x00000010, 0x00403062
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004400: 0x00000006, 0x00000000, 0x00010000, 0x0000a000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x00300c05
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004450: 0x00780504, 0x0111000d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004700: 0x00030002, 0x00100000, 0x00100010, 0x0b160000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004710: 0xffff4e50, 0x000ca924, 0x00100012, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004720: 0x00000000, 0x00000010, 0xf02c0000, 0xffff4e80
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004770: 0x000c0404, 0x00000042, 0x00000018, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004800: 0x188303fe, 0x00000000, 0x00000000, 0x00000120
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004810: 0x00000000, 0x0000000a, 0x02589574, 0x00001514
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004820: 0x00000089, 0x00000000, 0xf0330000, 0xffda0ea8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004840: 0x00000180, 0x00000180, 0x000e2200, 0x89423443
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004850: 0x284c49e6, 0x14d100e6, 0x87868786, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004860: 0x00000089, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004870: 0x00000000, 0x003e1820, 0x003e1820, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004890: 0x28090404, 0x00305400, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000048a0: 0x000f0010, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004900: 0x18030006, 0x00000003, 0x30000000, 0x00000120
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004910: 0x00000040, 0x00000003, 0x0000d57c, 0x00000018
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004920: 0x00000004, 0x02018000, 0xffde2201, 0x0f000030
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004930: 0xffed7201, 0x0f000030, 0xffe65001, 0x0f000030
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004940: 0xffcbce01, 0x0f000060, 0x33337070, 0xbcbc8080
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004950: 0xf0330000, 0xffde2230, 0xaf000000, 0x04030003
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004960: 0x00000000, 0xffed7230, 0xaf000000, 0x400000e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004970: 0x00028203, 0x00205400, 0x0000001c, 0x000000ff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004980: 0x00000004, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004990: 0x00000000, 0x00000000, 0xffe65030, 0xaf000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000049a0: 0x00f000db, 0x00000004, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000049b0: 0xff4c7a01, 0xffc99801, 0xffdaac01, 0xfff11a01
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000049c0: 0xffde2202, 0xffed7202, 0xffe65002, 0xffcbce02
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000049d0: 0xff4c7a02, 0xffc99802, 0xffdaac02, 0xfff11a02
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000049f0: 0xf0330000, 0xffcbce60, 0xaf000000, 0x0000ffff
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c00: 0x200003fe, 0x00000020, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c10: 0x0000003f, 0x00000000, 0x00000006, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c30: 0x00000000, 0x00000000, 0x00000072, 0x00000072
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c40: 0x00000000, 0xfffdffe0, 0x00010020, 0x00140020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c50: 0x3516c7ff, 0x078fff73, 0x374d49b2, 0x00785656
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005000: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08000898
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005020: 0x00000000, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005040: 0x00000000, 0x00000000, 0x08000a80, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08000898
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000050a0: 0x27bdffd8, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000050c0: 0x00000000, 0x00000000, 0x08000a80, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08000898
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005120: 0x8f4200d8, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005140: 0x00000000, 0x00000000, 0x08000a80, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005180: 0x00009800, 0x80008000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x080009e0
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000051a0: 0x8f634000, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000051c0: 0x00000000, 0x00000000, 0x08000cbc, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005200: 0x0800ff70, 0x00000000, 0x10400004, 0x0800ff98
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005210: 0x00000000, 0x00000000, 0x0800ff70, 0x40000040
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005220: 0x00641824, 0x10400004, 0x0800ff98, 0x0e000040
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005230: 0x00000000, 0x10400004, 0x0800ff98, 0x0800ff70
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005240: 0x40000040, 0x10400004, 0x0800ff98, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005250: 0x00000000, 0x10400004, 0x0800ff98, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005260: 0x0800ff70, 0xc0000000, 0x00000b50, 0xc0000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005270: 0x00000000, 0x0800ff98, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000898
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000052a0: 0x30420001, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000052c0: 0x00000000, 0x00000000, 0x08000a80, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x080009e8
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005320: 0x27bdffd8, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005340: 0x00000000, 0x00000000, 0x08000cbc, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005380: 0x00009800, 0x80000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08001240
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000053a0: 0x00000000, 0x00000000, 0x00000000, 0x40000020
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000053c0: 0x00000000, 0x00000000, 0x080008be, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005800: 0x00000000, 0x56000000, 0x00000000, 0xeb000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005810: 0x00000000, 0x9e000000, 0x00000000, 0xdb000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005820: 0x00000000, 0x99000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x0000013f
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005880: 0x00000000, 0x000003ff, 0x00000000, 0x000001b2
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005890: 0x00000000, 0x00000352, 0x00000000, 0x00000374
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005900: 0x00000000, 0x000000e8, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005980: 0x000000e8, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006000: 0x20000082, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006400: 0x00000000, 0x00000000, 0x00010091, 0xc0000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006410: 0x05000016, 0x05000016, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006430: 0x00000000, 0x14e41686, 0x168614e4, 0x01020000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000064c0: 0x00000005, 0x00000002, 0x00000122, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000064d0: 0x00000040, 0x00008d80, 0x00000000, 0x00d75e11
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000064f0: 0x00000002, 0x00000030, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006500: 0x03e10003, 0xb1e4723f, 0x000010dd, 0x00000003
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006550: 0x00000001, 0x02800000, 0x0000000f, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006560: 0x0000000f, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006800: 0x04130034, 0x20081082, 0x01009618, 0x4c39f69b
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006810: 0x01122010, 0xffffffff, 0x00000000, 0x00000040
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006880: 0x77fff020, 0x00000040, 0x80011686, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006890: 0x00800000, 0x88008000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006900: 0x1d6e9438, 0x00000000, 0x00000000, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00007000: 0x08000188, 0x00000000, 0x00000000, 0x000000c4
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00007010: 0xd1c00001, 0x02408200, 0x000500db, 0x03000a00
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00007030: 0x00010000, 0x000000c8, 0x000c0030, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0x00007500: 0x00000000, 0x00000000, 0x00000080, 0x00000000
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0: Host status block [00000001:00000056:(0000:0078:0000):(0000:00d1)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 0: NAPI info [00000056:00000056:(00e8:00d1:01ff):0000:(013f:0000:0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 1: Host status block [00000001:000000eb:(0000:0000:0000):(03ff:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 1: NAPI info [000000eb:000000eb:(0000:0000:01ff):03ff:(01ff:01ff:0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 2: Host status block [00000001:0000009e:(01b2:0000:0000):(0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 2: NAPI info [0000009e:0000009e:(0000:0000:01ff):01b2:(01b2:01b2:0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 3: Host status block [00000001:000000db:(0000:0000:0000):(0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 3: NAPI info [000000db:000000db:(0000:0000:01ff):0352:(0152:0152:0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 4: Host status block [00000001:00000099:(0000:0000:0374):(0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: 4: NAPI info [00000099:00000099:(0000:0000:01ff):0374:(0174:0174:0000:0000)]
Nov 22 20:08:16 iscsi03 kernel: tg3 0000:01:00.0 eth0: Link is down

Comment 33 Nathan Shearer 2019-11-23 07:05:06 UTC

Created attachment 286029 [details]
transmit timed out - Vanilla 5.3.11 with txo and gxo off

Comment 34 Hector Martin 2020-09-26 19:24:12 UTC

Having the same issues here on an iMac 5K (partno(BCM957766a) rev 57766001); the problem happens every ~10 minutes while rsyncing files to an NFS share over IPSec.

On a hunch, since I saw b7abee6ef8 (tg3: Apply short DMA frag workaround to 5906) fixed a similar issue on those devices, I tried applying that to 57766 too:

        if (tg3_flag(tp, 5755_PLUS) ||
            tg3_asic_rev(tp) == ASIC_REV_5906 ||
            tg3_asic_rev(tp) == ASIC_REV_57766) {
                tg3_flag_set(tp, SHORT_DMA_BUG);

And that seems to have helped; so far it hasn't crashed (though I'll call it for sure after a few days). Perhaps more tg3 devices need this bug workaround?

Comment 35 Hector Martin 2020-09-27 03:11:57 UTC

Looks like I spoke too soon, it still hung after a while with that patch. Oh well :(

Comment 36 Nathan Shearer 2020-09-27 04:43:11 UTC

This is a regression. The tg3 driver works fine on 4.4.203 on Mac Mini 6,1 and 6,2 computers, but will silently drop all packets after a tiny load on all newer kernel versions.

To reproduce:
 1) use a Mac Mini 6,1 or 6,2 or computer with the same network chip.
 2) ssh root@macmini dd if=/dev/urandom | dd of=/dev/null
 3) watch the bandwidh on the macmini. It will drop to 0b/s after a few moments

Mac Mini 6,2
  4.4.203 is not affected (12 hours, high cpu load, over 800GB transferred without error)
  4.9.203 is affected (eth0 timeout after only 6.3GB of transfer, no messages in dmesg)
  4.14.156 is affected (eth0 timeout in seconds after only 1.5GB of transfer, no messages in dmesg)
  5.3.11 is affected (eth0 timeout after only 6.3GB of transfer, no messages in dmesg)
  5.4 is affected (eth0 timeout after only 10GB of transfer, no messages in dmesg)

Changing tx rx sg and tso with ethtool does not help.

Comment 37 Nathan Shearer 2020-09-27 04:44:48 UTC

Created attachment 292675 [details]
mac-mini-6,2-C07JG6DZDWYN-dmidecode

mac-mini-6,2-C07JG6DZDWYN-dmidecode

Comment 38 Nathan Shearer 2020-09-27 04:45:24 UTC

Created attachment 292677 [details]
linux-4.4.203-config-working

linux-4.4.203-config-working

Comment 39 Nathan Shearer 2020-09-27 04:45:46 UTC

Created attachment 292679 [details]
linux-5.4-config-affected

linux-5.4-config-affected

Comment 40 Nathan Shearer 2020-09-27 05:58:08 UTC

*** Bug 205717 has been marked as a duplicate of this bug. ***

Comment 41 Hector Martin 2020-10-01 12:33:23 UTC

I've run the SSH test (in the tx direction, iMac -> outside) with all the offloading features enabled for several terabytes and it hasn't crashed yet, so it seems there is something more to reliably reproducing the problem than just spamming data out.

Comment 42 Dominik Mierzejewski 2020-12-11 11:43:56 UTC

Bug 203607 a duplicate?

Comment 43 Alexandros Kosiaris 2021-10-21 15:19:47 UTC

Meeting this as well recently in Macmini 7,1 running Debian Bullseye, kernel 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30).

This is a regression for me, it used to be running fine. The usual trigger is doing a VC in Google Meet. I 'll get at least 1 reset during a 30m meet.

I don't think a lot of traffic is the only trigger, plus it's not like my VC uses that much traffic (I have it at LD, 360p, not 720p both for outgoing and incoming and it was working flawlessly for years), it seems to require heavy CPU usage as well?

Comment 44 Pavan Chebbi 2021-10-29 08:51:42 UTC

Hi Alexandros, Can you please attach the dmesg that shows dump of the registers when the issue happens.

Comment 45 Tommy 2021-10-31 17:26:37 UTC

Same problem for me on a Mac Mini Late 2012 (NetXtreme BCM57766) and Debian 11 (5.10.0-9-amd64):

Let me know if you need more info @Pavan


[589372.776883] tg3 0000:01:00.0 enp1s0f0: transmit timed out, resetting
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000000: 0x168614e4, 0x00100406, 0x02000001, 0x00800040
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000010: 0xa040000c, 0x00000000, 0xa041000c, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000020: 0x00000000, 0x00000000, 0x00000000, 0x168614e4
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000100
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000040: 0x00000000, 0x02000000, 0xc8035001, 0x16002008
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x761b000f
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000070: 0x00001090, 0xff3f2eb3, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000080: 0x168614e4, 0xc00000d0, 0x00000000, 0x000001a0
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000090: 0x00000000, 0x0000019c, 0x00000000, 0x0000007d
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000000a0: 0x8005ac11, 0x00000002, 0x00000122, 0x00020010
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000000b0: 0x05908d80, 0x00105400, 0x00475c11, 0x10110142
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000000d0: 0x0000081f, 0x00000000, 0x00000000, 0x00010001
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000000f0: 0x00000000, 0x00000000, 0x00000000, 0x57766001
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000110: 0x00003100, 0x00002000, 0x000000a0, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000140: 0x66298093, 0x0000a820, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000001
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000160: 0x1b010002, 0x00000000, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000001b0: 0x00010018, 0x00000000, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000200: 0x00000000, 0x02000000, 0x00000000, 0x9d000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000210: 0x00000000, 0xfe000000, 0x00000000, 0x57000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000220: 0x00000000, 0xb7000000, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x0000007d
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000280: 0x00000000, 0x000001a0, 0x00000000, 0x000003ff
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000290: 0x00000000, 0x000003ba, 0x00000000, 0x0000005c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000300: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000310: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000320: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000330: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000340: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000350: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000360: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000370: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000380: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000390: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003a0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003b0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003c0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003d0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003e0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000003f0: 0x00000000, 0x0000019c, 0x00000000, 0x0000019c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000400: 0x00e04808, 0x02400000, 0x00001000, 0x00000880
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000410: 0x0000a820, 0x66298093, 0x0000a820, 0x66298093
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000420: 0x0000a820, 0x66298093, 0x0000a820, 0x66298093
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000430: 0x00000000, 0x00000000, 0x0000008c, 0x000005f2
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000460: 0x00000018, 0x00002620, 0x01ff0002, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000470: 0x80000000, 0x00000024, 0x00000000, 0x41200002
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000500: 0x00000008, 0x00000001, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000590: 0x00900110, 0x00000000, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x000005c0: 0xce9247d8, 0xac614464, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f0c
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00044, 0x34aac000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000670: 0xc6c72202, 0xdf379c59, 0xdc07a912, 0x55844f7d
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000680: 0x99e69f47, 0x055699cb, 0x7642fb08, 0x14057973
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000690: 0x990e5dda, 0x3ba17dce, 0x00000000, 0x00000000
[589375.868118] tg3 0000:01:00.0 enp1s0f0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[589375.895526] tg3 0000:01:00.0 enp1s0f0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[589375.895526] tg3 0000:01:00.0 enp1s0f0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[589375.895526] tg3 0000:01:00.0 enp1s0f0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[589375.895552] tg3 0000:01:00.0 enp1s0f0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[589375.895552] tg3 0000:01:00.0 enp1s0f0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[589375.895552] tg3 0000:01:00.0 enp1s0f0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[589375.895552] tg3 0000:01:00.0 enp1s0f0: 0x00000880: 0x00000000, 0x0002cb8f, 0x00000000, 0x00000000
[589375.895552] tg3 0000:01:00.0 enp1s0f0: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x002a0000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00000c80: 0x0000ea5c, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00000ce0: 0xff90bc02, 0x00000000, 0x0000002a, 0x00040028
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00000cf0: 0x00000000, 0x5000001c, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001000: 0x00000002, 0x00000000, 0xa0001212, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001010: 0x019c19c1, 0x00001212, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000001
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001440: 0x0000019c, 0x0000019c, 0x0000019c, 0x0000019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001450: 0x0000019c, 0x0000019c, 0x0000019c, 0x0000019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001460: 0x0000019c, 0x0000019c, 0x0000019c, 0x0000019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001470: 0x0000019c, 0x0000019c, 0x0000019c, 0x0000019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001800: 0x00000016, 0x00000000, 0x0000019c, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001850: 0x0000001f, 0x00000000, 0x000041b0, 0x019c019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001860: 0x01000000, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002010: 0x00000181, 0x00000001, 0x007bffff, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002100: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002110: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002120: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002130: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002140: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002150: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002160: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002170: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002180: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002190: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021a0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021b0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021c0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021d0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021e0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000021f0: 0x000a1d0e, 0x000da4d5, 0x00000060, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002200: 0x0002a272, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002250: 0x00000003, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002400: 0x00000012, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002450: 0x00000000, 0xffff0000, 0x02001800, 0x00006000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002470: 0x00000000, 0x000001c0, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000024c0: 0x02015101, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x00000032
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002c10: 0x00000000, 0x00000000, 0x00000019, 0x0000000c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00002c20: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x00000032
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003600: 0x00137400, 0x00130000, 0x00110000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003610: 0x00170000, 0x00130000, 0x00150000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003620: 0x00000000, 0x00000000, 0x00000000, 0x00022080
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003630: 0x00000000, 0x80008000, 0x00001080, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003640: 0x0000000a, 0x32f00003, 0x00000020, 0x00000019
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003650: 0x00000171, 0x00000bff, 0x57766001, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003660: 0x00000000, 0x00000000, 0x010400a7, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003670: 0x0000002a, 0xfeffff63, 0x0000000a, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003690: 0x80157766, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000036a0: 0x00000000, 0x00000000, 0x00000000, 0x00000044
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000036b0: 0x001003c8, 0x07ff07ff, 0x07ff07ff, 0x0100000d
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000036c0: 0xfffffffe, 0x008f2133, 0x00ed0634, 0x0000013d
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x000012df
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000036e0: 0x00000b19, 0x20800799, 0x00500799, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c30: 0x00000000, 0x00000000, 0x00000000, 0xffffa000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c50: 0x00000000, 0x000001b6, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003c80: 0x000001a0, 0x00000000, 0x000003ba, 0x0000005c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003cc0: 0x0000019c, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003d00: 0x00000000, 0xffff9000, 0x00000000, 0xffff8000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003d10: 0x00000000, 0xffff7000, 0x00000000, 0xffff6000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004000: 0x00000002, 0x00000000, 0x000a9e50, 0x0004e711
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004010: 0x00412660, 0x002d5012, 0x00800440, 0x01003822
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004030: 0x00000000, 0x01083620, 0x002fd010, 0x00459002
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004040: 0x00465000, 0x02142000, 0x00000010, 0x00412062
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004400: 0x00000006, 0x00000000, 0x00010000, 0x0000a000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x01204814
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004450: 0x02320427, 0x00fe00d7, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004700: 0x00030002, 0x00100000, 0x00100010, 0x1f650000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004710: 0xffff5990, 0x000c5914, 0x00100011, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004720: 0x00000000, 0x00000010, 0xf02c0000, 0xffff59c0
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004770: 0x000c0404, 0x00000042, 0x00000018, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004800: 0x188303fe, 0x00000000, 0x00000000, 0x00000120
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004810: 0x00000000, 0x0000000b, 0x05a89574, 0x00001514
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004820: 0x00000065, 0x00000000, 0xf0310000, 0xff37e0c0
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004840: 0x0000b33c, 0x0000b33c, 0x000e2200, 0x6542f638
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004850: 0xa0a704a0, 0x2a5e05ea, 0x59585958, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004860: 0x00000065, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004870: 0x00000000, 0x003e1820, 0x003e1820, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004890: 0x28090404, 0x00305400, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000048a0: 0x000f0010, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004900: 0x18030006, 0x00000003, 0x30000000, 0x00000120
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004910: 0x00000040, 0x00000003, 0x0000d574, 0x00000008
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004920: 0x00000013, 0x02090000, 0xff90bc01, 0x0f000030
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004930: 0xffcefc01, 0x0f000030, 0xfff3fc01, 0x0f000030
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004940: 0xfff5b001, 0x0f000030, 0xd0d0c6c6, 0x2b2bffff
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004950: 0xf0330000, 0xff90bc30, 0xaf000000, 0x13120012
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004960: 0x00000000, 0xffcefc30, 0xaf000000, 0x4000019c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004970: 0x00028203, 0x00205400, 0x0000001c, 0x000000ff
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004980: 0x00000013, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004990: 0x00000000, 0x00000000, 0xfff3fc30, 0xaf000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000049a0: 0x00f00049, 0x00000013, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000049b0: 0xffd94601, 0xfda79e01, 0xfda9f801, 0xffe70001
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000049c0: 0xff90bc02, 0xffcefc02, 0xfff3fc02, 0xfff5b002
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000049d0: 0xffd94602, 0xfda79e02, 0xfda9f802, 0xffe70002
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000049f0: 0xf0330000, 0xfff5b030, 0xaf000000, 0x0000ffff
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c00: 0x200003fe, 0x00000020, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c10: 0x0000003f, 0x00000000, 0x00000006, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c30: 0x00000000, 0x00680000, 0x00000020, 0x00000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c40: 0x00000000, 0xfffcffe0, 0x00010020, 0x00140020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c50: 0x3b9ffd9f, 0x1b6ffc5b, 0x45cee800, 0x55360202
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08000dcc
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005020: 0x1460fffb, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005040: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08000dc8
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000050a0: 0x00831821, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000050c0: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005100: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08000ddc
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005120: 0x1460fffb, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005140: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08000dd8
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000051a0: 0x00831821, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000051c0: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005200: 0x000003e8, 0x50f82688, 0x000003e8, 0x000003e8
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005210: 0x08000900, 0xc0000000, 0x000003e8, 0x0800ff68
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005220: 0x000003e8, 0x000003e8, 0x000003e8, 0x50f8259f
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005230: 0x00000000, 0xc0000000, 0xc0000000, 0x0800ff68
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005240: 0x000003e8, 0x000003e8, 0x000003e8, 0x000003e8
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005250: 0x000003e8, 0xc0000000, 0x0800ff68, 0x0800ff68
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005260: 0x000003e8, 0x50f8259f, 0x000003e8, 0x08000900
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005270: 0x50f826df, 0x000003e8, 0x000003e8, 0x50f826e7
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005280: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08000dcc
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000052a0: 0x00831821, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000052c0: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005300: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x08000dd4
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005320: 0x00831821, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005340: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08000dcc
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000053a0: 0x00831821, 0x00000000, 0x00000000, 0x40000020
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000053c0: 0x00000000, 0x00000000, 0x08000dde, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005800: 0x00000000, 0x02000000, 0x00000000, 0x9d000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005810: 0x00000000, 0xfe000000, 0x00000000, 0x57000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005820: 0x00000000, 0xb7000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x0000007d
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005880: 0x00000000, 0x000001a0, 0x00000000, 0x000003ff
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005890: 0x00000000, 0x000003ba, 0x00000000, 0x0000005c
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005900: 0x00000000, 0x0000019c, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005980: 0x0000019c, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006000: 0x20000082, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006400: 0x00000000, 0x00000000, 0x00010091, 0xc0000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006410: 0x05000016, 0x05000016, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006430: 0x00000000, 0x14e41686, 0x168614e4, 0x01020000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000064c0: 0x00000005, 0x00000002, 0x00000122, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000064d0: 0x00000040, 0x00008d80, 0x00000000, 0x00d75e11
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000064f0: 0x00000002, 0x00000030, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006500: 0x03e10003, 0x66298093, 0x0000a820, 0x00000003
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006550: 0x00000001, 0x02800000, 0x0000000f, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006560: 0x0000000f, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006800: 0x04130034, 0x20081082, 0x01009618, 0x5100759b
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006810: 0x01122010, 0xffffffff, 0x00000000, 0x00000040
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006880: 0x77fff020, 0x00000040, 0x80011686, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006890: 0x00800000, 0x88008000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006900: 0x9d92f7f8, 0x00000000, 0x00000000, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00007000: 0x08000188, 0x00000000, 0x00000000, 0x000000c4
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00007010: 0xd1c00001, 0x02408200, 0x000500db, 0x03000a00
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00007030: 0x00010000, 0x000000c8, 0x000c0030, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0x00007500: 0x00000000, 0x00000000, 0x00000080, 0x00000000
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0: Host status block [00000001:00000002:(0000:01b6:0000):(0000:0100)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 0: NAPI info [00000002:00000002:(019c:0100:01ff):0000:(007d:0000:0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 1: Host status block [00000001:0000009d:(0000:0000:0000):(01a0:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 1: NAPI info [0000009d:0000009d:(0000:0000:01ff):01a0:(01a0:01a0:0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 2: Host status block [00000001:000000fe:(03ff:0000:0000):(0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 2: NAPI info [000000fe:000000fe:(0000:0000:01ff):03ff:(01ff:01ff:0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 3: Host status block [00000001:00000057:(0000:0000:0000):(0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 3: NAPI info [00000057:00000057:(0000:0000:01ff):03ba:(01ba:01ba:0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 4: Host status block [00000001:000000b7:(0000:0000:005c):(0000:0000)]
[589375.895591] tg3 0000:01:00.0 enp1s0f0: 4: NAPI info [000000b7:000000b7:(0000:0000:01ff):005c:(005c:005c:0000:0000)]
[589375.925840] tg3 0000:01:00.0 enp1s0f0: Link is down
[589379.521213] tg3 0000:01:00.0 enp1s0f0: Link is up at 1000 Mbps, full duplex
[589379.521213] tg3 0000:01:00.0 enp1s0f0: Flow control is off for TX and off for RX
[589379.521213] tg3 0000:01:00.0 enp1s0f0: EEE is enabled

Comment 46 Alexandros Kosiaris 2021-11-01 17:06:07 UTC

Created attachment 299387 [details]
tg3_timing_out_akosiaris.dmesg

tg3 timing out dmesg

Comment 47 Alexandros Kosiaris 2021-11-01 17:07:09 UTC

(In reply to Pavan Chebbi from comment #44)
> Hi Alexandros, Can you please attach the dmesg that shows dump of the
> registers when the issue happens.

Done. Many thanks for looking into it Pavan!

Comment 48 Pavan Chebbi 2021-11-04 07:05:26 UTC

Hi Tommy, Alexandros,

In both your registers I see Write DMA Status Register (0x4c04) BIT 5 = Write DMA PCI Host Address Overflow Error is set.

Can you please check if LSO is enabled? If yes, can you disable LSO and test once?
Also please let me know what is the iommu status on your system.

Comment 49 Alexandros Kosiaris 2021-11-04 13:23:33 UTC

Hi Pavan,

If by LSO you mean TSO and GSO (which I think you do), yes both are enabled.

$ sudo ethtool --show-offload enp3s0f0 |grep offload
tcp-segmentation-offload: on
generic-segmentation-offload: on
...

I 'll disable them and test (the issue isn't readily reproducible, but I am hoping to have something to report by tomorrow)

As far as the iommu status question, I can report the following for dmesg output

[    0.059693] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.317860] iommu: Default domain type: Translated 
[    1.138970] pci 0000:00:00.0: Adding to iommu group 0
# sniping another 20 or so similar lines for other groups, let me know if you need them
[    1.139209] pci 0000:04:00.0: Adding to iommu group 20
[    1.154854] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    1.154854] AMD-Vi: AMD IOMMUv2 functionality not available on this system

Is that sufficient to answer your question? Or do you require something from under /sys/class/iommu/ ?

Comment 50 Tommy 2021-11-04 14:26:52 UTC

Good evening Pavan,
After my last comment I decieded to try the latest kernel from Debian Backports (5.14.0-0.bpo.2-amd64) but the problem was not solved. 

"cat /sys/class/iommu/dmar1/devices/$(PCI id of Ethernet Card)/enable" returns 1  so that probably means that IOMMU is active for the card?

ethtool says:

tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
macsec-hw-offload: off [fixed]
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

so I presume that LSO is enabled.
-----------------------------------------
After disabling LSO:

tcp-segmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
macsec-hw-offload: off [fixed]
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]


The problem persists, the IOMMU is still enabled.

Comment 51 Tommy 2021-11-04 18:53:20 UTC

I've also tried turning both TSO and GSO off:

tcp-segmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
macsec-hw-offload: off [fixed]
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]


but this does not solve the bug. 

@Alexandros try the SSH copy command someone mentioned earlier. I can reproduce the problem within a few seconds with it.

Comment 52 killerfoxi 2021-11-05 06:53:44 UTC

I've encountered the exact same problem on my macmini7,1 with the Debian kernel of the testing branch: 5.14 as well the one out of bullseye (5.10).

Exact same dmesg and symptoms, a little load trips it. Also tried turning off TSO and GSO, without success.

Happy to provide further information if needed, just let me know what. iommu is enabled as well.

Comment 53 Marian Rainer-Harbach 2021-11-06 15:07:11 UTC

I've started to encounter this issue on multiple HPE servers this year. Since I'm running Ubuntu 20.04 I created a report at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1950046 that also contains two example occurrences.

Comment 54 Pavan Chebbi 2021-11-08 09:11:13 UTC

Hi, is it possible to re-compile tg3 driver? 
Can you please add a print to show what is tp->pci_chip_rev_id ?

You may add a print inside tg3_reset_hw() function as below:

netdev_err(tp->dev, "tg3 chip rev = %x\n", tp->pci_chip_rev_id);

Thanks

Comment 55 killerfoxi 2021-11-08 10:19:42 UTC

Can't recompile right now, since the macmini is in active use for the next 4 days and I was adviced to not restart. In the meantime, does lspci not have the same info?

03:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe [14e4:1686] (rev 01)

And here more verbose (-vvvxxx):

03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 19
        IOMMU group: 12
        Region 0: Memory at a0700000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at a0710000 (64-bit, prefetchable) [size=64K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
                Read-only fields:
                        [PN] Part number: BCM957766a
                        [EC] Engineering changes: 106679-15
                        [SN] Serial number: 0123456789
                        [MN] Manufacture ID: 14e4
                        [RV] Reserved: checksum good, 26 byte(s) reserved
                Read/write fields:
                        [YA] Asset tag: XYZ01234567
                        [RW] Read-write area: 107 byte(s) free
                End
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000120
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number 00-00-38-c9-86-26-73-2a
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [1b0 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Kernel driver in use: tg3
        Kernel modules: tg3
00: e4 14 86 16 06 04 10 00 01 00 00 02 40 00 80 00
10: 0c 00 70 a0 00 00 00 00 0c 00 71 a0 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 e4 14 86 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 01 00 00
40: 00 00 00 00 01 00 00 00 01 50 03 c8 08 20 00 16
50: 03 58 fc 80 00 00 00 78 05 a0 86 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 98 02 01 f0 0f 00 1b 76
70: 90 10 00 00 b2 e2 f0 d4 00 00 00 00 00 00 00 00
80: e4 14 86 16 46 00 00 40 00 00 00 00 b5 02 00 00
90: 00 00 00 00 f0 01 00 00 00 00 00 00 af 00 00 00
a0: 11 ac 05 80 02 00 00 00 22 01 00 00 10 00 02 00
b0: 80 8d 90 05 00 54 10 00 11 5c 47 00 42 01 11 10
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 1f 08 00 00 00 04 00 00 00 00 00 00 01 00 01 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 01 60 76 57

Comment 56 Alexandros Kosiaris 2021-11-08 11:41:54 UTC

I see in /sys/module/tg3/drivers/pci:tg3/0000:03:00.0/revision 0x01 as a value if that helps. But I 'll give it a try and compile tg3.

Comment 57 killerfoxi 2021-11-08 12:36:39 UTC

Same revision here, when cat'ing that path.

Comment 58 Christoph Nelles 2022-01-01 04:36:18 UTC

I am having the same issue with my Dell Latitude 5495 notebook, under heavy load (remote: dd if=/dev/zero |nc ip-adresse 1025  notebook: nc -l -p 1025 > /dev/null)  the NIC crashes in seconds. 

I disabled the IOMMU and so far network is running stable. 

Can somebody try disabling the IOMMU at boot time. kernel parameter: amd_iommu=off intel_iommu=off and test if that changes anything?

Comment 59 Hector Martin 2022-01-01 04:56:10 UTC

I've been running with the IOMMU disabled for months and it has been mostly stable, I only remember it breaking once or twice. There's definitely something about the IOMMU that makes this problem happen.

Comment 61 hubsif 2022-01-02 09:18:33 UTC

I've set up Debian as home server on a Macmini7,1 recently (5.10.0-10-amd64) and ran into the same issues. I couldn't reproduce by heavy load though (ssh above), but streaming with tvheadend from a SAT>IP source wouldn't work longer than 15mins.

I disabled iommu two days ago by adding the kernel parameters from above (amd_iommu=off intel_iommu=off) and haven't had any issues since.

Comment 62 Pavan Chebbi 2022-01-02 13:00:44 UTC

Looks to me that in all the cases, we run into this issue of Write DMA host address overflow on 5776x devices https://bugzilla.kernel.org/show_bug.cgi?id=12877#c48

This is probably working ok when IOMMU is not used for the device since the IOVA is then is just physical memory address. There may be a related errata for this, I will try find it and update. The device itself is a very old one not available/sold anymore.

Comment 63 barzag 2022-05-11 10:31:02 UTC

hi, 

can you explain to me how to disable IOMMU on debian 10. I am newbie...
thank you

Comment 64 kousu 2022-06-24 14:54:17 UTC

I'm seeing the regression on a Macmini6,2 (late 2012), with a Broadcom BCM57766. The system was, as far as I know, fine under Ubuntu 20.04, but when I upgraded to Ubuntu 22.04 it immediately started having one, sometimes two or three 

>  tg3 0000:01:00.0 enp1s0f0: transmit timed out, resetting

per day, semi-randomly.

Here's uname and lspci:

root@johhny:~# uname -a
Linux johhny.example.com 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@johhny:~# lspci -nnvvvxxx -d 14e4:1686 
01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe [14e4:1686] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit Ethernet PCIe [14e4:1686]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 16
	IOMMU group: 12
	Region 0: Memory at a0400000 (64-bit, prefetchable) [size=64K]
	Region 2: Memory at a0410000 (64-bit, prefetchable) [size=64K]
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
		Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
		Read-only fields:
			[PN] Part number: BCM957766a
			[EC] Engineering changes: 106679-15
			[SN] Serial number: 0123456789
			[MN] Manufacture ID: 14e4
			[RV] Reserved: checksum good, 26 byte(s) reserved
		Read/write fields:
			[YA] Asset tag: XYZ01234567
			[RW] Read-write area: 107 byte(s) free
		End
	Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
		Vector table: BAR=2 offset=00000000
		PBA: BAR=2 offset=00000120
	Capabilities: [ac] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-11-18-b3
	Capabilities: [150 v1] Power Budgeting <?>
	Capabilities: [160 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [1b0 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: tg3
	Kernel modules: tg3
00: e4 14 86 16 06 04 10 00 01 00 00 02 40 00 80 00
10: 0c 00 40 a0 00 00 00 00 0c 00 41 a0 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 e4 14 86 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 01 00 00
40: 00 00 00 00 00 00 00 49 01 50 03 c8 08 20 00 16
50: 03 58 fc 80 00 00 00 78 05 a0 86 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 98 02 01 f0 0f 00 1b 76
70: 90 10 00 00 e7 bb 7f 96 00 00 00 00 00 00 00 00
80: e4 14 86 16 46 00 00 40 00 00 00 00 d3 01 00 00
90: 00 00 00 00 d3 00 00 00 00 00 00 00 18 01 00 00
a0: 11 ac 05 80 02 00 00 00 22 01 00 00 10 00 02 00
b0: 80 8d 90 05 00 54 10 00 11 5c 47 00 42 01 11 10
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 1f 08 00 00 00 00 00 00 00 00 00 00 01 00 01 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 01 60 76 57

And

root@johhny:~# cat /sys/module/tg3/drivers/pci\:tg3/0000\:01\:00.0/revision 
0x01

As above, I tried the decade-old workaround:

ethtool -K enp1s0f0 sg off
ethtool -K enp1s0f0 tso off
ethtool -K enp1s0f0 gso off

But I still had the connection drop. So that workaround isn't working anymore.


I am trying to IOMMU workaround now. Since this is Ubuntu with GRUB, I was able to apply it with:

root@johhny:~# echo 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX amd_iommu=off intel_iommu=off"' >  /etc/default/grub.d/disable_iommu.cfg
root@johhny:~# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/disable_iommu.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Sourcing file `/etc/default/grub.d/regdom.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.0-39-generic
Found initrd image: /boot/initrd.img-5.15.0-39-generic
Found linux image: /boot/vmlinuz-5.15.0-37-generic
Found initrd image: /boot/initrd.img-5.15.0-37-generic
Memtest86+ needs a 16-bit boot, that is not available on EFI, exiting
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
root@johhny:~# grep -ri intel_iommu /boot/grub/
/boot/grub/grub.cfg:	linux	/vmlinuz-5.15.0-39-generic root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
/boot/grub/grub.cfg:		linux	/vmlinuz-5.15.0-39-generic root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
/boot/grub/grub.cfg:		linux	/vmlinuz-5.15.0-39-generic root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
/boot/grub/grub.cfg:		linux	/vmlinuz-5.15.0-37-generic root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
/boot/grub/grub.cfg:		linux	/vmlinuz-5.15.0-37-generic root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
root@johhny:~# reboot

Hopefully seeing it in full helps you @barzag.

Comment 65 Andreas 2022-09-16 12:24:49 UTC

I have the same issue with two brand new DELL R6515 server.

Running with XCP-NG Citrix Xenserver Fork 8.2.1
DOM0 bast on CentOS 7.

[13:52 IT1XCP-NG-MASTER1 ~]# uname -a
Linux IT1XCP-NG-MASTER1 4.19.0+1 #1 SMP Thu Jan 13 12:55:45 CET 2022 x86_64 x86_64 x86_64 GNU/Linux

Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693559] ------------[ cut here ]------------
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693563] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693579] WARNING: CPU: 13 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x206/0x210
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693580] Modules linked in: arc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm fscache tun iscsi_tcp libiscsi_tcp drbd lru_cache bnx2fc(O) cnic(O) uio fcoe libfcoe libfc scsi_transport_fc openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 rdma_ucm nf_defrag_ipv4 libcrc32c iptable_filter ib_uverbs dm_multipath ib_umad rdma_cm ib_cm iw_cm sunrpc nls_iso8859_1 nls_cp437 vfat fat bnxt_re ib_core dm_mod crct10dif_pclmul crc32_pclmul dcdbas ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper sg i2c_piix4 ipmi_si ipmi_devintf
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693639]  ipmi_msghandler acpi_power_meter ip_tables x_tables sd_mod ahci libahci tg3 libata xhci_pci bnxt_en(O) xhci_hcd megaraid_sas(O) devlink scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_mod efivarfs ipv6 crc_ccitt
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693658] CPU: 13 PID: 0 Comm: swapper/13 Tainted: G           O      4.19.0+1 #1
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693660] Hardware name: Dell Inc. PowerEdge R6515/035YY8, BIOS 2.8.4 06/24/2022
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693663] RIP: e030:dev_watchdog+0x206/0x210
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693665] Code: 63 45 e0 eb 93 4c 89 e7 c6 05 ee 6c a7 00 01 e8 50 e1 fc ff 89 d9 48 89 c2 4c 89 e6 48 c7 c7 b8 fa e8 81 31 c0 e8 fa cc a0 ff <0f> 0b eb be 66 0f 1f 44 00 00 0f 1f 44 00 00 41 57 ba 18 00 00 00
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693666] RSP: e02b:ffff888280143ea8 EFLAGS: 00010286
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693668] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693669] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888280156420
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693670] RBP: ffff888272cc8480 R08: 0000000000000000 R09: 00000000000005ab
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693671] R10: 0000000000000004 R11: 0000000000000000 R12: ffff888272cc8000
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693672] R13: ffff88827119d580 R14: ffff888272cc845c R15: 000000000000000d
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693684] FS:  0000000000000000(0000) GS:ffff888280140000(0000) knlGS:0000000000000000
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693686] CS:  e033 DS: 002b ES: 002b CR0: 0000000080050033
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693687] CR2: 00007f2883b97000 CR3: 0000000128ef2000 CR4: 0000000000040660
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693690] Call Trace:
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693696]  <IRQ>
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693704]  ? pfifo_fast_enqueue+0xf0/0xf0
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693708]  call_timer_fn+0x2d/0x140
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693711]  run_timer_softirq+0x1e2/0x440
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693715]  ? handle_percpu_irq+0x37/0x50
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693718]  ? generic_handle_irq+0x24/0x30
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693722]  ? __evtchn_fifo_handle_events+0x180/0x1a0
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693725]  __do_softirq+0xd1/0x28c
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693730]  irq_exit+0xa8/0xc0
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693732]  xen_evtchn_do_upcall+0x2c/0x50
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693736]  xen_do_hypervisor_callback+0x29/0x40
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693737]  </IRQ>
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693741] RIP: e030:xen_hypercall_sched_op+0xa/0x20
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693742] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693743] RSP: e02b:ffffc9004011feb0 EFLAGS: 00000246
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693745] RAX: 0000000000000000 RBX: ffff888276e4d700 RCX: ffffffff810013aa
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693746] RDX: ffffffff8203d250 RSI: 0000000000000000 RDI: 0000000000000001
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693747] RBP: 000000000000000d R08: 0000000000000008 R09: 0000000000000000
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693748] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693749] R13: 0000000000000000 R14: ffff888276e4d700 R15: ffff888276e4d700
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693751]  ? xen_hypercall_sched_op+0xa/0x20
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693755]  ? xen_safe_halt+0xc/0x20
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693757]  ? default_idle+0x1a/0x140
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693760]  ? do_idle+0x1ea/0x260
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693762]  ? cpu_startup_entry+0x6f/0x80
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693764] ---[ end trace 05b4f36548e1d9b2 ]---
Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693769] tg3 0000:c1:00.0 eth0: transmit timed out, resetting
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951393] tg3 0000:c1:00.0 eth0: 0x00007030: 0x000e0000, 0x00004a50, 0x00170030, 0x00000000
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951397] tg3 0000:c1:00.0 eth0: 0: Host status block [00000001:000000b7:(0000:013d:0000):(0000:0017)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951401] tg3 0000:c1:00.0 eth0: 0: NAPI info [000000b7:000000b7:(0004:0017:01ff):0000:(020a:0000:0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951405] tg3 0000:c1:00.0 eth0: 1: Host status block [00000001:000000ea:(0000:0000:0000):(0063:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951408] tg3 0000:c1:00.0 eth0: 1: NAPI info [000000ea:000000ea:(0000:0000:01ff):0063:(0063:0063:0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951412] tg3 0000:c1:00.0 eth0: 2: Host status block [00000001:00000041:(07de:0000:0000):(0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951415] tg3 0000:c1:00.0 eth0: 2: NAPI info [00000041:00000041:(0000:0000:01ff):07de:(07de:07de:0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951419] tg3 0000:c1:00.0 eth0: 3: Host status block [00000001:00000058:(0000:0000:0000):(0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951422] tg3 0000:c1:00.0 eth0: 3: NAPI info [00000058:00000058:(0000:0000:01ff):0c12:(0412:0412:0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951425] tg3 0000:c1:00.0 eth0: 4: Host status block [00000001:000000b7:(0000:0000:04ef):(0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386626.951429] tg3 0000:c1:00.0 eth0: 4: NAPI info [000000b7:000000b7:(0000:0000:01ff):04ef:(04ef:04ef:0000:0000)]
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386627.053403] tg3 0000:c1:00.0: tg3_stop_block timed out, ofs=4c00 enable_bit=2
Sep 16 05:45:59 IT1XCP-NG-MASTER1 kernel: [386627.128139] tg3 0000:c1:00.0 eth0: Link is down
Sep 16 05:46:03 IT1XCP-NG-MASTER1 kernel: [386631.492362] tg3 0000:c1:00.0 eth0: Link is up at 1000 Mbps, full duplex
Sep 16 05:46:03 IT1XCP-NG-MASTER1 kernel: [386631.492365] tg3 0000:c1:00.0 eth0: Flow control is off for TX and off for RX
Sep 16 05:46:03 IT1XCP-NG-MASTER1 kernel: [386631.492367] tg3 0000:c1:00.0 eth0: EEE is disabled
Sep 16 05:46:08 IT1XCP-NG-MASTER1 kernel: [386636.701131] tg3 0000:c1:00.0 eth0: transmit timed out, resetting

[14:05 IT1XCP-NG-MASTER1 ~]# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 164f (rev 01)
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1650
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1651
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1652
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1653
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1654
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1655
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1656
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1657
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID Tri-Mode SAS3508 (rev 01)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
03:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
04:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
40:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
40:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 164f (rev 01)
40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
40:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
40:03.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
40:03.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
45:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
45:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
46:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
46:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
80:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
80:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 164f (rev 01)
80:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
80:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
80:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
80:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
80:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
83:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
83:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
84:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
84:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
85:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
c0:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
c0:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 164f (rev 01)
c0:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
c0:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
c0:03.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
c0:03.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
c0:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:05.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship PCIe GPP Bridge [1:0]
c0:05.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship PCIe GPP Bridge [1:0]
c0:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
c0:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
c0:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
c1:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
c1:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
c2:00.0 PCI bridge: PLDA PCI Express Bridge (rev 02)
c3:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
c8:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
c8:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
c9:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
c9:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA

eth0 is connected at

c1:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
        Subsystem: Dell PowerEdge R6515/R7515 LOM [1028:08fd]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 121
        Region 0: Memory at c7030000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at c7040000 (64-bit, prefetchable) [size=64K]
        Region 4: Memory at c7050000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at c7100000 [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: Broadcom NetXtreme Gigabit Ethernet
                Read-only fields:
                        [PN] Part number: BCM95720
                        [MN] Manufacture ID: 31 30 32 38
                        [V0] Vendor specific: FFV22.00.6
                        [V1] Vendor specific: DSV1028VPDR.VER1.0
                        [V2] Vendor specific: NPY2
                        [V3] Vendor specific: PMT1
                        [V4] Vendor specific: NMVBroadcom Corp
                        [V5] Vendor specific: DTINIC
                        [V6] Vendor specific: DCM1001008d452101008d45
                        [RV] Reserved: checksum good, 233 byte(s) reserved
                End
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [a0] MSI-X: Enable+ Count=17 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [13c v1] Device Serial Number 00-00-08-92-04-9d-bd-46
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Kernel driver in use: tg3
        Kernel modules: tg3
00: e4 14 5f 16 46 05 10 00 00 00 00 02 00 00 80 00
10: 0c 00 03 c7 00 00 00 00 0c 00 04 c7 00 00 00 00
20: 0c 00 05 c7 00 00 00 00 00 00 00 00 28 10 fd 08
30: 00 00 fc ff 48 00 00 00 00 00 00 00 ff 01 00 00
40: 00 00 00 00 00 00 00 a3 01 50 03 c8 08 20 00 64
50: 03 58 8c 81 00 00 00 78 05 a0 86 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 98 02 00 f0 81 00 38 00
70: b0 10 07 00 fe e9 ae f8 00 00 00 00 00 00 00 00
80: e4 14 5f 16 4b 00 00 00 00 00 00 00 7f 06 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 0b 03 00 00
a0: 11 ac 10 80 04 00 00 00 04 10 00 00 10 00 02 00
b0: 81 8d 00 10 2e 24 10 00 22 cc 04 00 40 00 12 10
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 1f 00 00 00 06 00 00 00 00 00 00 00 01 00 01 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 72 05 00 00 00 00 80 12 0a 60

Comment 66 Andreas 2022-09-20 12:45:18 UTC

(In reply to Andreas from comment #65)
> I have the same issue with two brand new DELL R6515 server.
> 
> Running with XCP-NG Citrix Xenserver Fork 8.2.1
> DOM0 bast on CentOS 7.
> 
> [13:52 IT1XCP-NG-MASTER1 ~]# uname -a
> Linux IT1XCP-NG-MASTER1 4.19.0+1 #1 SMP Thu Jan 13 12:55:45 CET 2022 x86_64
> x86_64 x86_64 GNU/Linux
> 
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693559] ------------[ cut
> here ]------------
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693563] NETDEV WATCHDOG:
> eth0 (tg3): transmit queue 0 timed out
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693579] WARNING: CPU: 13
> PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x206/0x210
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693580] Modules linked in:
> arc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm fscache tun
> iscsi_tcp libiscsi_tcp drbd lru_cache bnx2fc(O) cnic(O) uio fcoe libfcoe
> libfc scsi_transport_fc openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount
> nf_nat 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser
> libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp
> scsi_transport_srp ib_ipoib ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport
> xt_conntrack nf_conntrack nf_defrag_ipv6 rdma_ucm nf_defrag_ipv4 libcrc32c
> iptable_filter ib_uverbs dm_multipath ib_umad rdma_cm ib_cm iw_cm sunrpc
> nls_iso8859_1 nls_cp437 vfat fat bnxt_re ib_core dm_mod crct10dif_pclmul
> crc32_pclmul dcdbas ghash_clmulni_intel pcbc aesni_intel aes_x86_64
> crypto_simd cryptd glue_helper sg i2c_piix4 ipmi_si ipmi_devintf
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693639]  ipmi_msghandler
> acpi_power_meter ip_tables x_tables sd_mod ahci libahci tg3 libata xhci_pci
> bnxt_en(O) xhci_hcd megaraid_sas(O) devlink scsi_dh_rdac scsi_dh_hp_sw
> scsi_dh_emc scsi_dh_alua scsi_mod efivarfs ipv6 crc_ccitt
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693658] CPU: 13 PID: 0
> Comm: swapper/13 Tainted: G           O      4.19.0+1 #1
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693660] Hardware name:
> Dell Inc. PowerEdge R6515/035YY8, BIOS 2.8.4 06/24/2022
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693663] RIP:
> e030:dev_watchdog+0x206/0x210
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693665] Code: 63 45 e0 eb
> 93 4c 89 e7 c6 05 ee 6c a7 00 01 e8 50 e1 fc ff 89 d9 48 89 c2 4c 89 e6 48
> c7 c7 b8 fa e8 81 31 c0 e8 fa cc a0 ff <0f> 0b eb be 66 0f 1f 44 00 00 0f 1f
> 44 00 00 41 57 ba 18 00 00 00
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693666] RSP:
> e02b:ffff888280143ea8 EFLAGS: 00010286
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693668] RAX:
> 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693669] RDX:
> 0000000000000000 RSI: 0000000000000001 RDI: ffff888280156420
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693670] RBP:
> ffff888272cc8480 R08: 0000000000000000 R09: 00000000000005ab
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693671] R10:
> 0000000000000004 R11: 0000000000000000 R12: ffff888272cc8000
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693672] R13:
> ffff88827119d580 R14: ffff888272cc845c R15: 000000000000000d
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693684] FS: 
> 0000000000000000(0000) GS:ffff888280140000(0000) knlGS:0000000000000000
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693686] CS:  e033 DS: 002b
> ES: 002b CR0: 0000000080050033
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693687] CR2:
> 00007f2883b97000 CR3: 0000000128ef2000 CR4: 0000000000040660
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693690] Call Trace:
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693696]  <IRQ>
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693704]  ?
> pfifo_fast_enqueue+0xf0/0xf0
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693708] 
> call_timer_fn+0x2d/0x140
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693711] 
> run_timer_softirq+0x1e2/0x440
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693715]  ?
> handle_percpu_irq+0x37/0x50
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693718]  ?
> generic_handle_irq+0x24/0x30
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693722]  ?
> __evtchn_fifo_handle_events+0x180/0x1a0
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693725] 
> __do_softirq+0xd1/0x28c
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693730]  irq_exit+0xa8/0xc0
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693732] 
> xen_evtchn_do_upcall+0x2c/0x50
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693736] 
> xen_do_hypervisor_callback+0x29/0x40
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693737]  </IRQ>
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693741] RIP:
> e030:xen_hypercall_sched_op+0xa/0x20
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693742] Code: 51 41 53 b8
> 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
> cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc
> cc cc cc cc cc cc cc cc cc cc
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693743] RSP:
> e02b:ffffc9004011feb0 EFLAGS: 00000246
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693745] RAX:
> 0000000000000000 RBX: ffff888276e4d700 RCX: ffffffff810013aa
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693746] RDX:
> ffffffff8203d250 RSI: 0000000000000000 RDI: 0000000000000001
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693747] RBP:
> 000000000000000d R08: 0000000000000008 R09: 0000000000000000
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693748] R10:
> 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693749] R13:
> 0000000000000000 R14: ffff888276e4d700 R15: ffff888276e4d700
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693751]  ?
> xen_hypercall_sched_op+0xa/0x20
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693755]  ?
> xen_safe_halt+0xc/0x20
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693757]  ?
> default_idle+0x1a/0x140
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693760]  ?
> do_idle+0x1ea/0x260
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693762]  ?
> cpu_startup_entry+0x6f/0x80
> Sep 16 05:45:57 IT1XCP-NG-MASTER1 kernel: [386625.693764] ---[ end trace
> 05b4f36548e1d9b2 ]---

Problem raised by this xenserver bug:

https://bugs.xenserver.org/projects/XSO/issues/XSO-1021?filter=allopenissues

Comment 67 Andreas 2022-09-20 13:51:47 UTC

(In reply to Andreas from comment #66)

> Problem raised by this xenserver bug:
> 
> https://bugs.xenserver.org/projects/XSO/issues/XSO-1021?filter=allopenissues

Sorry, I mixed up another bug with the reported "tg3" problem.

Please disregard oer delete Comment 65 (and 66).

Comment 68 Andreas 2022-10-24 12:40:43 UTC

It's really a serious problem, we can't use our new servers productively:

today again the same error with driver tg3:

[code]
[14:37 IT1XCP-NG-SLAVE1 ~]# head -100 /var/log/kern.log.1
Oct 24 04:34:04 IT1XCP-NG-SLAVE1 kernel: [234547.162607] CIFS VFS: Server 192.168.1.202 has not responded in 120 seconds. Reconnecting...
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862213] ------------[ cut here ]------------
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862241] NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862261] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x206/0x210
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862262] Modules linked in: tun drbd lru_cache arc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm fscache iscsi_tcp libiscsi_tcp bnx2fc(O) cnic(O) uio fcoe libfcoe libfc scsi_transport_fc openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ipt_REJECT nf_reject_ipv4 ib_uverbs xt_tcpudp xt_multiport dm_multipath xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ib_umad iptable_filter rdma_cm ib_cm iw_cm hid_generic usbhid hid sunrpc nls_iso8859_1 nls_cp437 vfat fat bnxt_re dm_mod dcdbas ib_core crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper sg
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862340]  i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ip_tables x_tables sd_mod ahci libahci tg3 libata xhci_pci bnxt_en(O) xhci_hcd megaraid_sas(O) devlink scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_mod efivarfs ipv6 crc_ccitt
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862366] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      4.19.0+1 #1
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862367] Hardware name: Dell Inc. PowerEdge R6515/035YY8, BIOS 2.8.5 08/18/2022
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862370] RIP: e030:dev_watchdog+0x206/0x210
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862373] Code: 63 45 e0 eb 93 4c 89 e7 c6 05 ee 6c a7 00 01 e8 50 e1 fc ff 89 d9 48 89 c2 4c 89 e6 48 c7 c7 b8 fa e8 81 31 c0 e8 fa cc a0 ff <0f> 0b eb be 66 0f 1f 44 00 00 0f 1f 44 00 00 41 57 ba 18 00 00 00
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862374] RSP: e02b:ffff88827fe03ea8 EFLAGS: 00010286
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862376] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862377] RDX: 0000000000000001 RSI: ffff88827fe03db4 RDI: 0000000000000004
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862379] RBP: ffff88827407c480 R08: ffff888276cbfa00 R09: ffff888276800d90
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862380] R10: 0000000000000004 R11: 0000000000000000 R12: ffff88827407c000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862381] R13: ffff888271018d80 R14: ffff88827407c45c R15: 0000000000000000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862394] FS:  0000000000000000(0000) GS:ffff88827fe00000(0000) knlGS:0000000000000000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862395] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862396] CR2: 00000000006ec0f8 CR3: 000000026f540000 CR4: 0000000000040660
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862398] Call Trace:
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862401]  <IRQ>
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862407]  ? pfifo_fast_enqueue+0xf0/0xf0
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862411]  call_timer_fn+0x2d/0x140
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862414]  run_timer_softirq+0x1e2/0x440
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862418]  ? handle_percpu_irq+0x37/0x50
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862421]  ? generic_handle_irq+0x24/0x30
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862425]  ? __evtchn_fifo_handle_events+0x180/0x1a0
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862428]  __do_softirq+0xd1/0x28c
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862433]  irq_exit+0xa8/0xc0
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862435]  xen_evtchn_do_upcall+0x2c/0x50
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862438]  xen_do_hypervisor_callback+0x29/0x40
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862440]  </IRQ>
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862444] RIP: e030:xen_hypercall_sched_op+0xa/0x20
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862446] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862447] RSP: e02b:ffffffff82003e58 EFLAGS: 00000246
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862448] RAX: 0000000000000000 RBX: ffffffff82011740 RCX: ffffffff810013aa
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862449] RDX: ffffffff8203d250 RSI: 0000000000000000 RDI: 0000000000000001
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862450] RBP: 0000000000000000 R08: 00000000aafc1bea R09: 0000000000000000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862451] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862452] R13: 0000000000000000 R14: ffffffff82011740 R15: ffffffff82011740
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862455]  ? xen_hypercall_sched_op+0xa/0x20
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862458]  ? xen_safe_halt+0xc/0x20
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862461]  ? default_idle+0x1a/0x140
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862464]  ? do_idle+0x1ea/0x260
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862466]  ? cpu_startup_entry+0x6f/0x80
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862471]  ? start_kernel+0x558/0x578
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862473]  ? set_init_arg+0x55/0x55
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862475]  ? xen_start_kernel+0x599/0x5a3
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862477] ---[ end trace 6ae276d8e2e1b0d4 ]---
Oct 24 04:38:40 IT1XCP-NG-SLAVE1 kernel: [234822.862500] tg3 0000:c1:00.0 eth0: transmit timed out, resetting
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118114] tg3 0000:c1:00.0 eth0: 0x00000000: 0x165f14e4, 0x00100546, 0x02000000, 0x00800000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118141] tg3 0000:c1:00.0 eth0: 0x00000010: 0xc703000c, 0x00000000, 0xc704000c, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118145] tg3 0000:c1:00.0 eth0: 0x00000020: 0xc705000c, 0x00000000, 0x00000000, 0x08fd1028
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118159] tg3 0000:c1:00.0 eth0: 0x00000030: 0xfffc0000, 0x00000048, 0x00000000, 0x0000010b
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118163] tg3 0000:c1:00.0 eth0: 0x00000040: 0x00000000, 0xd5000000, 0xc8035001, 0x64002008
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118166] tg3 0000:c1:00.0 eth0: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118172] tg3 0000:c1:00.0 eth0: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118176] tg3 0000:c1:00.0 eth0: 0x00000070: 0x000710b0, 0xffbd3bf8, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118185] tg3 0000:c1:00.0 eth0: 0x00000080: 0x165f14e4, 0x800000f8, 0x00000000, 0x0000047e
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118195] tg3 0000:c1:00.0 eth0: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000007cf
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118201] tg3 0000:c1:00.0 eth0: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118208] tg3 0000:c1:00.0 eth0: 0x000000b0: 0x10008d81, 0x0010242e, 0x0004cc22, 0x10120040
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118219] tg3 0000:c1:00.0 eth0: 0x000000d0: 0x0000001f, 0x00000006, 0x00000000, 0x00010001
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118229] tg3 0000:c1:00.0 eth0: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x600a1280
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118233] tg3 0000:c1:00.0 eth0: 0x00000100: 0x13c10001, 0x00000000, 0x00010000, 0x000ef030
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118241] tg3 0000:c1:00.0 eth0: 0x00000110: 0x00002000, 0x000031c0, 0x000000a0, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118248] tg3 0000:c1:00.0 eth0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118256] tg3 0000:c1:00.0 eth0: 0x00000140: 0x049dab26, 0x00000892, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118262] tg3 0000:c1:00.0 eth0: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000001
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118272] tg3 0000:c1:00.0 eth0: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118279] tg3 0000:c1:00.0 eth0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118287] tg3 0000:c1:00.0 eth0: 0x00000200: 0x00000000, 0xd5000000, 0x00000000, 0x9e000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118290] tg3 0000:c1:00.0 eth0: 0x00000210: 0x00000000, 0x47000000, 0x00000000, 0xd3000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118296] tg3 0000:c1:00.0 eth0: 0x00000220: 0x00000000, 0x79000000, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118301] tg3 0000:c1:00.0 eth0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000007cf
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118304] tg3 0000:c1:00.0 eth0: 0x00000280: 0x00000000, 0x0000047e, 0x00000000, 0x000001dc
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118309] tg3 0000:c1:00.0 eth0: 0x00000290: 0x00000000, 0x00000475, 0x00000000, 0x00000438
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118313] tg3 0000:c1:00.0 eth0: 0x00000300: 0x00000000, 0x000000ea, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118318] tg3 0000:c1:00.0 eth0: 0x00000400: 0x18e04808, 0x00400000, 0x00001000, 0x00000880
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118322] tg3 0000:c1:00.0 eth0: 0x00000410: 0x00000892, 0x049dab26, 0x00000892, 0x049dab26
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118331] tg3 0000:c1:00.0 eth0: 0x00000420: 0x00000892, 0x049dab26, 0x00000892, 0x049dab26
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118338] tg3 0000:c1:00.0 eth0: 0x00000430: 0x00000400, 0x00000000, 0x000000bc, 0x000005f2
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118349] tg3 0000:c1:00.0 eth0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x082a3800
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118353] tg3 0000:c1:00.0 eth0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000102
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118357] tg3 0000:c1:00.0 eth0: 0x00000460: 0x00000008, 0x00002620, 0x01ff0102, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118362] tg3 0000:c1:00.0 eth0: 0x00000470: 0x80000000, 0x00000000, 0x10000000, 0x40000042
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118366] tg3 0000:c1:00.0 eth0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118373] tg3 0000:c1:00.0 eth0: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118397] tg3 0000:c1:00.0 eth0: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118404] tg3 0000:c1:00.0 eth0: 0x000005b0: 0x00000000, 0x00000008, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118409] tg3 0000:c1:00.0 eth0: 0x000005c0: 0x6a93bf90, 0xa30dfa42, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118414] tg3 0000:c1:00.0 eth0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f0c
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118418] tg3 0000:c1:00.0 eth0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x60000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118424] tg3 0000:c1:00.0 eth0: 0x00000620: 0x00000446, 0x00000000, 0x00000000, 0x00000000
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118430] tg3 0000:c1:00.0 eth0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118435] tg3 0000:c1:00.0 eth0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118442] tg3 0000:c1:00.0 eth0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118454] tg3 0000:c1:00.0 eth0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118464] tg3 0000:c1:00.0 eth0: 0x00000670: 0xe4403cde, 0x6d7cad27, 0x0c4dbd8d, 0x421244d6
Oct 24 04:38:41 IT1XCP-NG-SLAVE1 kernel: [234824.118467] tg3 0000:c1:00.0 eth0: 0x00000680: 0xd4103ef3, 0xcb41ef69, 0xbae1396c, 0xe5f51df9
[/code]

Comment 69 barzag 2022-11-28 11:22:30 UTC

(In reply to kousu from comment #64)
> I'm seeing the regression on a Macmini6,2 (late 2012), with a Broadcom
> BCM57766. The system was, as far as I know, fine under Ubuntu 20.04, but
> when I upgraded to Ubuntu 22.04 it immediately started having one, sometimes
> two or three 
> 
> >  tg3 0000:01:00.0 enp1s0f0: transmit timed out, resetting
> 
> per day, semi-randomly.
> 
> Here's uname and lspci:
> 
> root@johhny:~# uname -a
> Linux johhny.example.com 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9 23:42:32
> UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> 
> root@johhny:~# lspci -nnvvvxxx -d 14e4:1686 
> 01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme
> BCM57766 Gigabit Ethernet PCIe [14e4:1686] (rev 01)
>       Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit
> Ethernet PCIe [14e4:1686]
>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 256 bytes
>       Interrupt: pin A routed to IRQ 16
>       IOMMU group: 12
>       Region 0: Memory at a0400000 (64-bit, prefetchable) [size=64K]
>       Region 2: Memory at a0410000 (64-bit, prefetchable) [size=64K]
>       Capabilities: [48] Power Management version 3
>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>       Capabilities: [50] Vital Product Data
>               Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
>               Read-only fields:
>                       [PN] Part number: BCM957766a
>                       [EC] Engineering changes: 106679-15
>                       [SN] Serial number: 0123456789
>                       [MN] Manufacture ID: 14e4
>                       [RV] Reserved: checksum good, 26 byte(s) reserved
>               Read/write fields:
>                       [YA] Asset tag: XYZ01234567
>                       [RW] Read-write area: 107 byte(s) free
>               End
>       Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>               Address: 0000000000000000  Data: 0000
>       Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
>               Vector table: BAR=2 offset=00000000
>               PBA: BAR=2 offset=00000120
>       Capabilities: [ac] Express (v2) Endpoint, MSI 00
>               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1
> <64us
>                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> SlotPowerLimit 10.000W
>               DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                       RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
>                       MaxPayload 128 bytes, MaxReadReq 4096 bytes
>               DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+
> TransPend-
>               LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit
> Latency L0s
> <2us, L1 <64us
>                       ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>               LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
>                       ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>               LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
>                       TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP-
> LTR+
>                        10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt-
> EETLPPrefix-
>                        EmergencyPowerReduction Not Supported,
> EmergencyPowerReductionInit-
>                        FRS- TPHComp- ExtTPHComp-
>                        AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR-
> OBFF Disabled,
>                        AtomicOpsCtl: ReqEn-
>               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>                        Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance-
> ComplianceSOS-
>                        Compliance De-emphasis: -6dB
>               LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete-
> EqualizationPhase1-
>                        EqualizationPhase2- EqualizationPhase3-
> LinkEqualizationRequest-
>                        Retimer- 2Retimers- CrosslinkRes: unsupported
>       Capabilities: [100 v1] Advanced Error Reporting
>               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP-
> ECRC- UnsupReq- ACSViol-
>               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP-
> ECRC- UnsupReq- ACSViol-
>               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+
> ECRC- UnsupReq- ACSViol-
>               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr+
>               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr+
>               AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn-
> ECRCChkCap+
> ECRCChkEn-
>                       MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>               HeaderLog: 00000000 00000000 00000000 00000000
>       Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-11-18-b3
>       Capabilities: [150 v1] Power Budgeting <?>
>       Capabilities: [160 v1] Virtual Channel
>               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>               Arb:    Fixed- WRR32- WRR64- WRR128-
>               Ctrl:   ArbSelect=Fixed
>               Status: InProgress-
>               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>                       Status: NegoPending- InProgress-
>       Capabilities: [1b0 v1] Latency Tolerance Reporting
>               Max snoop latency: 0ns
>               Max no snoop latency: 0ns
>       Kernel driver in use: tg3
>       Kernel modules: tg3
> 00: e4 14 86 16 06 04 10 00 01 00 00 02 40 00 80 00
> 10: 0c 00 40 a0 00 00 00 00 0c 00 41 a0 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 e4 14 86 16
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 01 00 00
> 40: 00 00 00 00 00 00 00 49 01 50 03 c8 08 20 00 16
> 50: 03 58 fc 80 00 00 00 78 05 a0 86 00 00 00 00 00
> 60: 00 00 00 00 00 00 00 00 98 02 01 f0 0f 00 1b 76
> 70: 90 10 00 00 e7 bb 7f 96 00 00 00 00 00 00 00 00
> 80: e4 14 86 16 46 00 00 40 00 00 00 00 d3 01 00 00
> 90: 00 00 00 00 d3 00 00 00 00 00 00 00 18 01 00 00
> a0: 11 ac 05 80 02 00 00 00 22 01 00 00 10 00 02 00
> b0: 80 8d 90 05 00 54 10 00 11 5c 47 00 42 01 11 10
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 1f 08 00 00 00 00 00 00 00 00 00 00 01 00 01 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 01 60 76 57
> 
> And
> 
> root@johhny:~# cat /sys/module/tg3/drivers/pci\:tg3/0000\:01\:00.0/revision 
> 0x01
> 
> As above, I tried the decade-old workaround:
> 
> ethtool -K enp1s0f0 sg off
> ethtool -K enp1s0f0 tso off
> ethtool -K enp1s0f0 gso off
> 
> But I still had the connection drop. So that workaround isn't working
> anymore.
> 
> 
> I am trying to IOMMU workaround now. Since this is Ubuntu with GRUB, I was
> able to apply it with:
> 
> root@johhny:~# echo 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX amd_iommu=off
> intel_iommu=off"' >  /etc/default/grub.d/disable_iommu.cfg
> root@johhny:~# update-grub
> Sourcing file `/etc/default/grub'
> Sourcing file `/etc/default/grub.d/disable_iommu.cfg'
> Sourcing file `/etc/default/grub.d/init-select.cfg'
> Sourcing file `/etc/default/grub.d/regdom.cfg'
> Generating grub configuration file ...
> Found linux image: /boot/vmlinuz-5.15.0-39-generic
> Found initrd image: /boot/initrd.img-5.15.0-39-generic
> Found linux image: /boot/vmlinuz-5.15.0-37-generic
> Found initrd image: /boot/initrd.img-5.15.0-37-generic
> Memtest86+ needs a 16-bit boot, that is not available on EFI, exiting
> Warning: os-prober will not be executed to detect other bootable partitions.
> Systems on them will not be added to the GRUB boot configuration.
> Check GRUB_DISABLE_OS_PROBER documentation entry.
> done
> root@johhny:~# grep -ri intel_iommu /boot/grub/
> /boot/grub/grub.cfg:  linux   /vmlinuz-5.15.0-39-generic
> root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-39-generic
> root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-39-generic
> root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset
> dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
> /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-37-generic
> root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-37-generic
> root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset
> dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
> root@johhny:~# reboot
> 
> Hopefully seeing it in full helps you @barzag.

Thank you so much kousu that works !!!!! You are the best

Comment 70 Hakan K.Yilmaz 2023-04-10 20:01:26 UTC

(In reply to barzag from comment #69)
> (In reply to kousu from comment #64)
> > I'm seeing the regression on a Macmini6,2 (late 2012), with a Broadcom
> > BCM57766. The system was, as far as I know, fine under Ubuntu 20.04, but
> > when I upgraded to Ubuntu 22.04 it immediately started having one,
> sometimes
> > two or three 
> > 
> > >  tg3 0000:01:00.0 enp1s0f0: transmit timed out, resetting
> > 
> > per day, semi-randomly.
> > 
> > Here's uname and lspci:
> > 
> > root@johhny:~# uname -a
> > Linux johhny.example.com 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9
> 23:42:32
> > UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > root@johhny:~# lspci -nnvvvxxx -d 14e4:1686 
> > 01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries
> NetXtreme
> > BCM57766 Gigabit Ethernet PCIe [14e4:1686] (rev 01)
> >       Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM57766 Gigabit
> > Ethernet PCIe [14e4:1686]
> >       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR- FastB2B- DisINTx+
> >       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort-
> > <MAbort- >SERR- <PERR- INTx-
> >       Latency: 0, Cache Line Size: 256 bytes
> >       Interrupt: pin A routed to IRQ 16
> >       IOMMU group: 12
> >       Region 0: Memory at a0400000 (64-bit, prefetchable) [size=64K]
> >       Region 2: Memory at a0410000 (64-bit, prefetchable) [size=64K]
> >       Capabilities: [48] Power Management version 3
> >               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> > PME(D0+,D1-,D2-,D3hot+,D3cold+)
> >               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> >       Capabilities: [50] Vital Product Data
> >               Product Name: Broadcom NetXtreme Gigabit Ethernet Controller
> >               Read-only fields:
> >                       [PN] Part number: BCM957766a
> >                       [EC] Engineering changes: 106679-15
> >                       [SN] Serial number: 0123456789
> >                       [MN] Manufacture ID: 14e4
> >                       [RV] Reserved: checksum good, 26 byte(s) reserved
> >               Read/write fields:
> >                       [YA] Asset tag: XYZ01234567
> >                       [RW] Read-write area: 107 byte(s) free
> >               End
> >       Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
> >               Address: 0000000000000000  Data: 0000
> >       Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
> >               Vector table: BAR=2 offset=00000000
> >               PBA: BAR=2 offset=00000120
> >       Capabilities: [ac] Express (v2) Endpoint, MSI 00
> >               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us,
> L1
> > <64us
> >                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> > SlotPowerLimit 10.000W
> >               DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >                       RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
> >                       MaxPayload 128 bytes, MaxReadReq 4096 bytes
> >               DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+
> > TransPend-
> >               LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit
> > Latency L0s
> > <2us, L1 <64us
> >                       ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> >               LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
> >                       ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
> >               LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> >                       TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> NROPrPrP-
> > LTR+
> >                        10BitTagComp- 10BitTagReq- OBFF Not Supported,
> ExtFmt-
> > EETLPPrefix-
> >                        EmergencyPowerReduction Not Supported,
> > EmergencyPowerReductionInit-
> >                        FRS- TPHComp- ExtTPHComp-
> >                        AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR-
> > OBFF Disabled,
> >                        AtomicOpsCtl: ReqEn-
> >               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
> SpeedDis-
> >                        Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance-
> > ComplianceSOS-
> >                        Compliance De-emphasis: -6dB
> >               LnkSta2: Current De-emphasis Level: -3.5dB,
> > EqualizationComplete-
> > EqualizationPhase1-
> >                        EqualizationPhase2- EqualizationPhase3-
> > LinkEqualizationRequest-
> >                        Retimer- 2Retimers- CrosslinkRes: unsupported
> >       Capabilities: [100 v1] Advanced Error Reporting
> >               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP-
> > ECRC- UnsupReq- ACSViol-
> >               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP-
> > ECRC- UnsupReq- ACSViol-
> >               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF+ MalfTLP+
> > ECRC- UnsupReq- ACSViol-
> >               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr+
> >               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr+
> >               AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn-
> > ECRCChkCap+
> > ECRCChkEn-
> >                       MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >               HeaderLog: 00000000 00000000 00000000 00000000
> >       Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-11-18-b3
> >       Capabilities: [150 v1] Power Budgeting <?>
> >       Capabilities: [160 v1] Virtual Channel
> >               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> >               Arb:    Fixed- WRR32- WRR64- WRR128-
> >               Ctrl:   ArbSelect=Fixed
> >               Status: InProgress-
> >               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> >                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> >                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> >                       Status: NegoPending- InProgress-
> >       Capabilities: [1b0 v1] Latency Tolerance Reporting
> >               Max snoop latency: 0ns
> >               Max no snoop latency: 0ns
> >       Kernel driver in use: tg3
> >       Kernel modules: tg3
> > 00: e4 14 86 16 06 04 10 00 01 00 00 02 40 00 80 00
> > 10: 0c 00 40 a0 00 00 00 00 0c 00 41 a0 00 00 00 00
> > 20: 00 00 00 00 00 00 00 00 00 00 00 00 e4 14 86 16
> > 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 01 00 00
> > 40: 00 00 00 00 00 00 00 49 01 50 03 c8 08 20 00 16
> > 50: 03 58 fc 80 00 00 00 78 05 a0 86 00 00 00 00 00
> > 60: 00 00 00 00 00 00 00 00 98 02 01 f0 0f 00 1b 76
> > 70: 90 10 00 00 e7 bb 7f 96 00 00 00 00 00 00 00 00
> > 80: e4 14 86 16 46 00 00 40 00 00 00 00 d3 01 00 00
> > 90: 00 00 00 00 d3 00 00 00 00 00 00 00 18 01 00 00
> > a0: 11 ac 05 80 02 00 00 00 22 01 00 00 10 00 02 00
> > b0: 80 8d 90 05 00 54 10 00 11 5c 47 00 42 01 11 10
> > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > d0: 1f 08 00 00 00 00 00 00 00 00 00 00 01 00 01 00
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > f0: 00 00 00 00 00 00 00 00 00 00 00 00 01 60 76 57
> > 
> > And
> > 
> > root@johhny:~# cat /sys/module/tg3/drivers/pci\:tg3/0000\:01\:00.0/revision 
> > 0x01
> > 
> > As above, I tried the decade-old workaround:
> > 
> > ethtool -K enp1s0f0 sg off
> > ethtool -K enp1s0f0 tso off
> > ethtool -K enp1s0f0 gso off
> > 
> > But I still had the connection drop. So that workaround isn't working
> > anymore.
> > 
> > 
> > I am trying to IOMMU workaround now. Since this is Ubuntu with GRUB, I was
> > able to apply it with:
> > 
> > root@johhny:~# echo 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX amd_iommu=off
> > intel_iommu=off"' >  /etc/default/grub.d/disable_iommu.cfg
> > root@johhny:~# update-grub
> > Sourcing file `/etc/default/grub'
> > Sourcing file `/etc/default/grub.d/disable_iommu.cfg'
> > Sourcing file `/etc/default/grub.d/init-select.cfg'
> > Sourcing file `/etc/default/grub.d/regdom.cfg'
> > Generating grub configuration file ...
> > Found linux image: /boot/vmlinuz-5.15.0-39-generic
> > Found initrd image: /boot/initrd.img-5.15.0-39-generic
> > Found linux image: /boot/vmlinuz-5.15.0-37-generic
> > Found initrd image: /boot/initrd.img-5.15.0-37-generic
> > Memtest86+ needs a 16-bit boot, that is not available on EFI, exiting
> > Warning: os-prober will not be executed to detect other bootable
> partitions.
> > Systems on them will not be added to the GRUB boot configuration.
> > Check GRUB_DISABLE_OS_PROBER documentation entry.
> > done
> > root@johhny:~# grep -ri intel_iommu /boot/grub/
> > /boot/grub/grub.cfg:  linux   /vmlinuz-5.15.0-39-generic
> > root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> > intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> > /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-39-generic
> > root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> > intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> > /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-39-generic
> > root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset
> > dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
> > /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-37-generic
> > root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro  amd_iommu=off
> > intel_iommu=off cfg80211.ieee80211_regdom=CA quiet splash $vt_handoff
> > /boot/grub/grub.cfg:          linux   /vmlinuz-5.15.0-37-generic
> > root=UUID=e4816d82-5634-4d30-8d22-b3108bf9054a ro single nomodeset
> > dis_ucode_ldr  amd_iommu=off intel_iommu=off cfg80211.ieee80211_regdom=CA
> > root@johhny:~# reboot
> > 
> > Hopefully seeing it in full helps you @barzag.
> 
> Thank you so much kousu that works !!!!! You are the best


I had the same issue with my Macmini6,2 running:

Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
Broadcom BCM4331

...
Mar 15 16:15:54 munich kernel: tg3 0000:01:00.0 enp1s0f0: transmit timed out, resetting
Mar 15 16:15:57 munich kernel: tg3 0000:01:00.0 enp1s0f0: 0x00000000: 0x168614e4, 0x00100406, 0x02000001, 0x00800040
...

Adding intel_iommu=on iommu=pt helped in my case.

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"


After updating grub and rebooting the network interface is stable now.

Note You need to log in before you can comment on or make changes to this bug.

akosiaris
alan
alexanderyt
anahi_mf2
andreasjantz
andrew.brockbank
barzaghi.julien
bastienphilbert
carnil
dominik
egorenar-dev
goran
graham.gear
hakan
hubsif
info+kernel
jdavis
jgudtmann
jrussell_88
kenyon
kernel
killerfoxi
krissn
mail
marian+kernel
mchan
net_xplorer
nick
pavan.chebbi
phodamentals
rgouveia+kernelorg
richard
szg00000
ysangkok