Bug 10767 - Seg Fault Instead of Swapping
Summary: Seg Fault Instead of Swapping
Status: RESOLVED CODE_FIX
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-05-21 02:45 UTC by Brian Vowell
Modified: 2008-09-16 11:47 UTC (History)
0 users

See Also:
Kernel Version: 2.6.25.4
Subsystem:
Regression: ---
Bisected commit-id:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Brian Vowell 2008-05-21 02:45:58 UTC 
May 20 13:11:00 ishtar kernel: ------------[ cut here ]------------
May 20 13:11:00 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0xd2b/0x191f()
May 20 13:11:00 ishtar kernel: Modules linked in: fuse dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util jedec_probe button cfi_probe gen_probe k8temp ck804xrom hwmon mtd usb_storage chipreg serio_raw map_funcs i2c_nforce2 pcspkr i2c_core sg pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last unloaded: scsi_wait_scan]
May 20 13:11:00 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 20 13:11:00 ishtar kernel:
May 20 13:11:00 ishtar kernel: Call Trace:
May 20 13:11:00 ishtar kernel:  <IRQ>  [<ffffffff81034481>] warn_on_slowpath+0x51/0x63
May 20 13:11:00 ishtar kernel:  [<ffffffff81023577>] kernel_map_pages+0xe8/0xed
May 20 13:11:00 ishtar kernel:  [<ffffffff8103ca36>] lock_timer_base+0x26/0x4c
May 20 13:11:00 ishtar kernel:  [<ffffffff812baa38>] tcp_snd_test+0x17/0xdf
May 20 13:11:00 ishtar kernel:  [<ffffffff812bab4d>] tcp_may_send_now+0x4d/0x5a
May 20 13:11:00 ishtar kernel:  [<ffffffff812b68ce>] tcp_ack+0xd2b/0x191f
May 20 13:11:00 ishtar kernel:  [<ffffffff812b9dbc>] tcp_rcv_established+0xe2/0x8ce
May 20 13:11:00 ishtar kernel:  [<ffffffff812bffba>] tcp_v4_do_rcv+0x2c2/0x48b
May 20 13:11:00 ishtar kernel:  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
May 20 13:11:00 ishtar kernel:  [<ffffffff812ae376>] __inet_lookup_established+0xdf/0x17b
May 20 13:11:00 ishtar kernel:  [<ffffffff812c1d10>] tcp_v4_rcv+0x6a3/0x700
May 20 13:11:00 ishtar kernel:  [<ffffffff812a7558>] ip_local_deliver+0xd4/0x18e
May 20 13:11:00 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 20 13:11:00 ishtar kernel:  [<ffffffff8128a2b7>] netif_receive_skb+0x351/0x372
May 20 13:11:00 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 20 13:11:00 ishtar kernel:  [<ffffffff8128c2ce>] net_rx_action+0xb6/0x1bf
May 20 13:11:00 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 20 13:11:00 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 20 13:11:00 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 20 13:11:00 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 20 13:11:00 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 20 13:11:00 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 20 13:11:00 ishtar kernel:  <EOI>  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 20 13:11:00 ishtar kernel:  [<ffffffff8101c138>] lapic_next_event+0x0/0xa
May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 20 13:11:00 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 20 13:11:00 ishtar kernel:
May 20 13:11:00 ishtar kernel: ---[ end trace f970ab335f443aca ]---
Comment 1 Anonymous Emailer 2008-05-21 10:57:58 UTC 
Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 21 May 2008 02:45:59 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=10767
> 
>            Summary: Seg Fault Instead of Swapping

Strange description?

>            Product: Networking
>            Version: 2.5
>      KernelVersion: 2.6.25.4
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: IPV4
>         AssignedTo: shemminger@linux-foundation.org
>         ReportedBy: brian.vowell@gmail.com
> 
> 
> May 20 13:11:00 ishtar kernel: ------------[ cut here ]------------
> May 20 13:11:00 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:2539
> tcp_ack+0xd2b/0x191f()

In 2.6.25 this is

	if (WARN_ON(!tp->sacked_out && tp->fackets_out))
		tp->fackets_out = 0;

> May 20 13:11:00 ishtar kernel: Modules linked in: fuse dm_mirror dm_multipath
> dm_mod cfi_cmdset_0002 cfi_util jedec_probe button cfi_probe gen_probe k8temp
> ck804xrom hwmon mtd usb_storage chipreg serio_raw map_funcs i2c_nforce2
> pcspkr
> i2c_core sg pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod
> scsi_mod
> raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb
> ehci_hcd [last unloaded: scsi_wait_scan]
> May 20 13:11:00 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
> May 20 13:11:00 ishtar kernel:
> May 20 13:11:00 ishtar kernel: Call Trace:
> May 20 13:11:00 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
> warn_on_slowpath+0x51/0x63
> May 20 13:11:00 ishtar kernel:  [<ffffffff81023577>]
> kernel_map_pages+0xe8/0xed
> May 20 13:11:00 ishtar kernel:  [<ffffffff8103ca36>]
> lock_timer_base+0x26/0x4c
> May 20 13:11:00 ishtar kernel:  [<ffffffff812baa38>] tcp_snd_test+0x17/0xdf
> May 20 13:11:00 ishtar kernel:  [<ffffffff812bab4d>]
> tcp_may_send_now+0x4d/0x5a
> May 20 13:11:00 ishtar kernel:  [<ffffffff812b68ce>] tcp_ack+0xd2b/0x191f
> May 20 13:11:00 ishtar kernel:  [<ffffffff812b9dbc>]
> tcp_rcv_established+0xe2/0x8ce
> May 20 13:11:00 ishtar kernel:  [<ffffffff812bffba>]
> tcp_v4_do_rcv+0x2c2/0x48b
> May 20 13:11:00 ishtar kernel:  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
> May 20 13:11:00 ishtar kernel:  [<ffffffff812ae376>]
> __inet_lookup_established+0xdf/0x17b
> May 20 13:11:00 ishtar kernel:  [<ffffffff812c1d10>] tcp_v4_rcv+0x6a3/0x700
> May 20 13:11:00 ishtar kernel:  [<ffffffff812a7558>]
> ip_local_deliver+0xd4/0x18e
> May 20 13:11:00 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
> May 20 13:11:00 ishtar kernel:  [<ffffffff8128a2b7>]
> netif_receive_skb+0x351/0x372
> May 20 13:11:00 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
> May 20 13:11:00 ishtar kernel:  [<ffffffff8128c2ce>] net_rx_action+0xb6/0x1bf
> May 20 13:11:00 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
> May 20 13:11:00 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
> May 20 13:11:00 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> May 20 13:11:00 ishtar kernel:  [<ffffffff8101c138>] lapic_next_event+0x0/0xa
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 20 13:11:00 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> May 20 13:11:00 ishtar kernel:
> May 20 13:11:00 ishtar kernel: ---[ end trace f970ab335f443aca ]---
Comment 2 Brian Vowell 2008-05-21 11:40:46 UTC 
Please forgive my lack of technical skills-- I am not a developer.

The bug report referenced below recently occurred on a pretty busy file
server of mine running Fedora 8 with a custom 2.6.25.4 kernel.  The attached
hardware RAID controller presents 12 LUNs, which were striped using the
Linux mdadm tools, and then the Linux LVM was used to create a single
logical volume.  The filesystem used is XFS.

Since upgrading from 2.6.25.3 to 2.6.25.4, my log has shown a large number
of errors that I don't possess the skills to diagnose which component is
actually causing the errors.  I can verify however, that the hardware is
good.  I have run memtest86 and the vendor's diagnostic tools and verified
that everything is running well.

The errors in /var/log/messages look like this:
May 19 06:30:30 ishtar kernel: Call Trace:
May 19 06:30:30 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 06:30:30 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 06:30:30 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 06:30:30 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 06:30:30 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 06:30:30 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 06:30:30 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 06:30:30 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 06:30:30 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 06:30:30 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 06:30:30 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 06:30:30 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 06:30:30 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 06:30:30 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 19 06:30:30 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 19 06:30:30 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a

The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor, 4GB
of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual
software RAID SATA disks for boot and swap.

The call trace included with the bug report is the only one that appears in
the log with the "cut here" lines.




On 5/21/08, Andrew Morton <akpm@linux-foundation.org> wrote:
>
>
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Wed, 21 May 2008 02:45:59 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.orgwrote:
>
> > http://bugzilla.kernel.org/show_bug.cgi?id=10767
> >
> >            Summary: Seg Fault Instead of Swapping
>
> Strange description?
>
> >            Product: Networking
> >            Version: 2.5
> >      KernelVersion: 2.6.25.4
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: IPV4
> >         AssignedTo: shemminger@linux-foundation.org
> >         ReportedBy: brian.vowell@gmail.com
> >
> >
> > May 20 13:11:00 ishtar kernel: ------------[ cut here ]------------
> > May 20 13:11:00 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:2539
> > tcp_ack+0xd2b/0x191f()
>
> In 2.6.25 this is
>
>         if (WARN_ON(!tp->sacked_out && tp->fackets_out))
>                 tp->fackets_out = 0;
>
> > May 20 13:11:00 ishtar kernel: Modules linked in: fuse dm_mirror
> dm_multipath
> > dm_mod cfi_cmdset_0002 cfi_util jedec_probe button cfi_probe gen_probe
> k8temp
> > ck804xrom hwmon mtd usb_storage chipreg serio_raw map_funcs i2c_nforce2
> pcspkr
> > i2c_core sg pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod
> scsi_mod
> > raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb
> > ehci_hcd [last unloaded: scsi_wait_scan]
> > May 20 13:11:00 ishtar kernel: Pid: 0, comm: swapper Not tainted
> 2.6.25.4 #1
> > May 20 13:11:00 ishtar kernel:
> > May 20 13:11:00 ishtar kernel: Call Trace:
> > May 20 13:11:00 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
> > warn_on_slowpath+0x51/0x63
> > May 20 13:11:00 ishtar kernel:  [<ffffffff81023577>]
> kernel_map_pages+0xe8/0xed
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8103ca36>]
> lock_timer_base+0x26/0x4c
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812baa38>]
> tcp_snd_test+0x17/0xdf
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812bab4d>]
> tcp_may_send_now+0x4d/0x5a
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812b68ce>] tcp_ack+0xd2b/0x191f
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812b9dbc>]
> > tcp_rcv_established+0xe2/0x8ce
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812bffba>]
> tcp_v4_do_rcv+0x2c2/0x48b
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8129a07a>]
> __qdisc_run+0xf6/0x1c8
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812ae376>]
> > __inet_lookup_established+0xdf/0x17b
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812c1d10>]
> tcp_v4_rcv+0x6a3/0x700
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812a7558>]
> > ip_local_deliver+0xd4/0x18e
> > May 20 13:11:00 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8128a2b7>]
> > netif_receive_skb+0x351/0x372
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8128c2ce>]
> net_rx_action+0xb6/0x1bf
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8103919b>]
> __do_softirq+0x65/0xce
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100ce9c>]
> call_softirq+0x1c/0x28
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
> > May 20 13:11:00 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100c221>]
> ret_from_intr+0x0/0xa
> > May 20 13:11:00 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
> > default_idle+0x0/0x55
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8101c138>]
> lapic_next_event+0x0/0xa
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf9>]
> default_idle+0x31/0x55
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100adf4>]
> default_idle+0x2c/0x55
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> > May 20 13:11:00 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> > May 20 13:11:00 ishtar kernel:
> > May 20 13:11:00 ishtar kernel: ---[ end trace f970ab335f443aca ]---
>
>
Please forgive my lack of technical skills-- I am not a developer.<br><br>The bug report referenced below recently occurred on a pretty busy file server of mine running Fedora 8 with a custom <a href="http://2.6.25.4">2.6.25.4</a> kernel.&nbsp; The attached hardware RAID controller presents 12 LUNs, which were striped using the Linux mdadm tools, and then the Linux LVM was used to create a single logical volume.&nbsp; The filesystem used is XFS.&nbsp; <br>
<br>Since upgrading from <a href="http://2.6.25.3">2.6.25.3</a> to <a href="http://2.6.25.4">2.6.25.4</a>, my log has shown a large number of errors that I don&#39;t possess the skills to diagnose which component is actually causing the errors.&nbsp; I can verify however, that the hardware is good.&nbsp; I have run memtest86 and the vendor&#39;s diagnostic tools and verified that everything is running well.<br>
<br>The errors in /var/log/messages look like this:<br>May 19 06:30:30 ishtar kernel: Call Trace:<br>May 19 06:30:30 ishtar kernel:&nbsp; &lt;IRQ&gt;&nbsp; [&lt;ffffffff81075d35&gt;] __alloc_pages+0x33f/0x35a<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff810934a4&gt;] new_slab+0x3f/0x242<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff81093bba&gt;] __slab_alloc+0x212/0x43b<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8128654c&gt;] __netdev_alloc_skb+0x29/0x43<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8128654c&gt;] __netdev_alloc_skb+0x29/0x43<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff81094bd6&gt;] __kmalloc_node_track_caller+0x75/0xab<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8128584a&gt;] __alloc_skb+0x6a/0x133<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8128654c&gt;] __netdev_alloc_skb+0x29/0x43<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8123a405&gt;] tg3_alloc_rx_skb+0xc4/0x153<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff81241343&gt;] tg3_poll+0x3be/0x7df<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8128c2ce&gt;] net_rx_action+0xb6/0x1bf<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8103919b&gt;] __do_softirq+0x65/0xce<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100ce9c&gt;] call_softirq+0x1c/0x28<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100e544&gt;] do_softirq+0x2c/0x68<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff810390f2&gt;] irq_exit+0x3f/0x83<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100c221&gt;] ret_from_intr+0x0/0xa<br>May 19 06:30:30 ishtar kernel:&nbsp; &lt;EOI&gt;&nbsp; [&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8101c138&gt;] lapic_next_event+0x0/0xa<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100adf9&gt;] default_idle+0x31/0x55<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100adf4&gt;] default_idle+0x2c/0x55<br>
May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>May 19 06:30:30 ishtar kernel:&nbsp; [&lt;ffffffff8100ae94&gt;] cpu_idle+0x77/0x9a<br><br>The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor, 4GB of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual software RAID SATA disks for boot and swap.<br>
<br>The call trace included with the bug report is the only one that appears in the log with the &quot;cut here&quot; lines.<br><br><br><br><br><div><span class="gmail_quote">On 5/21/08, <b class="gmail_sendername">Andrew Morton</b> &lt;<a href="mailto:akpm@linux-foundation.org">akpm@linux-foundation.org</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br> (switched to email.&nbsp;&nbsp;Please respond via emailed reply-to-all, not via the<br> bugzilla web interface).<br> <br> On Wed, 21 May 2008 02:45:59 -0700 (PDT) <a href="mailto:bugme-daemon@bugzilla.kernel.org">bugme-daemon@bugzilla.kernel.org</a> wrote:<br>
 <br> &gt; <a href="http://bugzilla.kernel.org/show_bug.cgi?id=10767">http://bugzilla.kernel.org/show_bug.cgi?id=10767</a><br> &gt;<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Summary: Seg Fault Instead of Swapping<br> <br> Strange description?<br>
 <br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Product: Networking<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Version: 2.5<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KernelVersion: <a href="http://2.6.25.4">2.6.25.4</a><br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Platform: All<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; OS/Version: Linux<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Tree: Mainline<br>
 &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Status: NEW<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Severity: normal<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Priority: P1<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Component: IPV4<br> &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AssignedTo: <a href="mailto:shemminger@linux-foundation.org">shemminger@linux-foundation.org</a><br>
 &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ReportedBy: <a href="mailto:brian.vowell@gmail.com">brian.vowell@gmail.com</a><br> &gt;<br> &gt;<br> &gt; May 20 13:11:00 ishtar kernel: ------------[ cut here ]------------<br> &gt; May 20 13:11:00 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:2539<br>
 &gt; tcp_ack+0xd2b/0x191f()<br> <br> In 2.6.25 this is<br> <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (WARN_ON(!tp-&gt;sacked_out &amp;&amp; tp-&gt;fackets_out))<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tp-&gt;fackets_out = 0;<br> <br> &gt; May 20 13:11:00 ishtar kernel: Modules linked in: fuse dm_mirror dm_multipath<br>
 &gt; dm_mod cfi_cmdset_0002 cfi_util jedec_probe button cfi_probe gen_probe k8temp<br> &gt; ck804xrom hwmon mtd usb_storage chipreg serio_raw map_funcs i2c_nforce2 pcspkr<br> &gt; i2c_core sg pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod scsi_mod<br>
 &gt; raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb<br> &gt; ehci_hcd [last unloaded: scsi_wait_scan]<br> &gt; May 20 13:11:00 ishtar kernel: Pid: 0, comm: swapper Not tainted <a href="http://2.6.25.4">2.6.25.4</a> #1<br>
 &gt; May 20 13:11:00 ishtar kernel:<br> &gt; May 20 13:11:00 ishtar kernel: Call Trace:<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;&lt;IRQ&gt;&nbsp;&nbsp;[&lt;ffffffff81034481&gt;]<br> &gt; warn_on_slowpath+0x51/0x63<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff81023577&gt;] kernel_map_pages+0xe8/0xed<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8103ca36&gt;] lock_timer_base+0x26/0x4c<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812baa38&gt;] tcp_snd_test+0x17/0xdf<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812bab4d&gt;] tcp_may_send_now+0x4d/0x5a<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812b68ce&gt;] tcp_ack+0xd2b/0x191f<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812b9dbc&gt;]<br> &gt; tcp_rcv_established+0xe2/0x8ce<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812bffba&gt;] tcp_v4_do_rcv+0x2c2/0x48b<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8129a07a&gt;] __qdisc_run+0xf6/0x1c8<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812ae376&gt;]<br> &gt; __inet_lookup_established+0xdf/0x17b<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812c1d10&gt;] tcp_v4_rcv+0x6a3/0x700<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812a7558&gt;]<br> &gt; ip_local_deliver+0xd4/0x18e<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff812a7b0d&gt;] ip_rcv+0x4fb/0x53a<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128a2b7&gt;]<br>
 &gt; netif_receive_skb+0x351/0x372<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8124150d&gt;] tg3_poll+0x588/0x7df<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128c2ce&gt;] net_rx_action+0xb6/0x1bf<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8103919b&gt;] __do_softirq+0x65/0xce<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100ce9c&gt;] call_softirq+0x1c/0x28<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100e544&gt;] do_softirq+0x2c/0x68<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff810390f2&gt;] irq_exit+0x3f/0x83<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100c221&gt;] ret_from_intr+0x0/0xa<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;&lt;EOI&gt;&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;]<br> &gt; default_idle+0x0/0x55<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8101c138&gt;] lapic_next_event+0x0/0xa<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adf9&gt;] default_idle+0x31/0x55<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adf4&gt;] default_idle+0x2c/0x55<br> &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>
 &gt; May 20 13:11:00 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100ae94&gt;] cpu_idle+0x77/0x9a<br> &gt; May 20 13:11:00 ishtar kernel:<br> &gt; May 20 13:11:00 ishtar kernel: ---[ end trace f970ab335f443aca ]---<br> <br> </blockquote>
</div><br>
Comment 3 Anonymous Emailer 2008-05-21 11:53:56 UTC 
Reply-To: akpm@linux-foundation.org

On Wed, 21 May 2008 11:40:43 -0700 "Brian Vowell" <brian.vowell@gmail.com> wrote:

> Please forgive my lack of technical skills-- I am not a developer.

Thank you for reporting this kernel bug!

> The bug report referenced below recently occurred on a pretty busy file
> server of mine running Fedora 8 with a custom 2.6.25.4 kernel.  The attached
> hardware RAID controller presents 12 LUNs, which were striped using the
> Linux mdadm tools, and then the Linux LVM was used to create a single
> logical volume.  The filesystem used is XFS.
> 
> Since upgrading from 2.6.25.3 to 2.6.25.4, my log has shown a large number
> of errors that I don't possess the skills to diagnose which component is
> actually causing the errors.  I can verify however, that the hardware is
> good.  I have run memtest86 and the vendor's diagnostic tools and verified
> that everything is running well.
> 
> The errors in /var/log/messages look like this:
> May 19 06:30:30 ishtar kernel: Call Trace:
> May 19 06:30:30 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
> __alloc_pages+0x33f/0x35a
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> May 19 06:30:30 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
> May 19 06:30:30 ishtar kernel:  [<ffffffff81093bba>]
> __slab_alloc+0x212/0x43b
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff81094bd6>]
> __kmalloc_node_track_caller+0x75/0xab
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff8123a405>]
> tg3_alloc_rx_skb+0xc4/0x153
> May 19 06:30:30 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128c2ce>]
> net_rx_action+0xb6/0x1bf
> May 19 06:30:30 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
> May 19 06:30:30 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
> May 19 06:30:30 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8101c138>]
> lapic_next_event+0x0/0xa
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> 
> The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor, 4GB
> of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual
> software RAID SATA disks for boot and swap.
> 
> The call trace included with the bug report is the only one that appears in
> the log with the "cut here" lines.
> 

OK, that is the TCP code performing self-checks and finding an
unexpected state.  

I was just a bit concerned about the "segfault" part of the
description.  Because often when the kernel crashes this appears to the
operator as a userspace segfault.  But the above trace isn't a kernel
crash - it's just a warning and the kernel does attept to fix things up
in this case.

If however you were really seeing unexpected segfaults then it could be
the case that the kernel is crashing.  But I assume this is not the
case, and that the text "Oops" does not appear in the logs.
Comment 4 Brian Vowell 2008-05-21 12:18:13 UTC 
No, the kernel didn't fully crash with an oops error.  What does happen
during the time when the errors show up is that there is usually a fair bit
of I/O going on, and then the segfault errors start streaming through the
logs.  During the time that the errors show up, both network and I/O will
freeze and the machine will be completely unresponsive for 5-10 seconds-- no
new logins, no HTTP responses or anything.

Again, to my knowledge, this did not occur with 2.6.25.3, but I only ran
that kernel for a short time, and it's possible that the error occurred but
was not noticed.


--b

On 5/21/08, Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 21 May 2008 11:40:43 -0700 "Brian Vowell" <brian.vowell@gmail.com>
> wrote:
>
> > Please forgive my lack of technical skills-- I am not a developer.
>
>
> Thank you for reporting this kernel bug!
>
>
> > The bug report referenced below recently occurred on a pretty busy file
> > server of mine running Fedora 8 with a custom 2.6.25.4 kernel.  The
> attached
> > hardware RAID controller presents 12 LUNs, which were striped using the
> > Linux mdadm tools, and then the Linux LVM was used to create a single
> > logical volume.  The filesystem used is XFS.
> >
> > Since upgrading from 2.6.25.3 to 2.6.25.4, my log has shown a large
> number
> > of errors that I don't possess the skills to diagnose which component is
> > actually causing the errors.  I can verify however, that the hardware is
> > good.  I have run memtest86 and the vendor's diagnostic tools and
> verified
> > that everything is running well.
> >
> > The errors in /var/log/messages look like this:
> > May 19 06:30:30 ishtar kernel: Call Trace:
> > May 19 06:30:30 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
> > __alloc_pages+0x33f/0x35a
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> > May 19 06:30:30 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
> > May 19 06:30:30 ishtar kernel:  [<ffffffff81093bba>]
> > __slab_alloc+0x212/0x43b
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> > __netdev_alloc_skb+0x29/0x43
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> > __netdev_alloc_skb+0x29/0x43
> > May 19 06:30:30 ishtar kernel:  [<ffffffff81094bd6>]
> > __kmalloc_node_track_caller+0x75/0xab
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8128584a>]
> __alloc_skb+0x6a/0x133
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> > __netdev_alloc_skb+0x29/0x43
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8123a405>]
> > tg3_alloc_rx_skb+0xc4/0x153
> > May 19 06:30:30 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8128c2ce>]
> > net_rx_action+0xb6/0x1bf
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8103919b>]
> __do_softirq+0x65/0xce
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100ce9c>]
> call_softirq+0x1c/0x28
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
> > May 19 06:30:30 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100c221>]
> ret_from_intr+0x0/0xa
> > May 19 06:30:30 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
> > default_idle+0x0/0x55
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8101c138>]
> > lapic_next_event+0x0/0xa
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf9>]
> default_idle+0x31/0x55
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf4>]
> default_idle+0x2c/0x55
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> > May 19 06:30:30 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> >
> > The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor,
> 4GB
> > of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual
> > software RAID SATA disks for boot and swap.
> >
> > The call trace included with the bug report is the only one that appears
> in
> > the log with the "cut here" lines.
> >
>
>
> OK, that is the TCP code performing self-checks and finding an
> unexpected state.
>
> I was just a bit concerned about the "segfault" part of the
> description.  Because often when the kernel crashes this appears to the
> operator as a userspace segfault.  But the above trace isn't a kernel
> crash - it's just a warning and the kernel does attept to fix things up
> in this case.
>
> If however you were really seeing unexpected segfaults then it could be
> the case that the kernel is crashing.  But I assume this is not the
> case, and that the text "Oops" does not appear in the logs.
>
No, the kernel didn&#39;t fully crash with an oops error.&nbsp; What does happen during the time when the errors show up is that there is usually a fair bit of I/O going on, and then the segfault errors start streaming through the logs.&nbsp; During the time that the errors show up, both network and I/O will freeze and the machine will be completely unresponsive for 5-10 seconds-- no new logins, no HTTP responses or anything.<br>
<br>Again, to my knowledge, this did not occur with <a href="http://2.6.25.3">2.6.25.3</a>, but I only ran that kernel for a short time, and it&#39;s possible that the error occurred but was not noticed.<br><br><br>--b<br>
<br><div><span class="gmail_quote">On 5/21/08, <b class="gmail_sendername">Andrew Morton</b> &lt;<a href="mailto:akpm@linux-foundation.org">akpm@linux-foundation.org</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Wed, 21 May 2008 11:40:43 -0700 &quot;Brian Vowell&quot; &lt;<a href="mailto:brian.vowell@gmail.com">brian.vowell@gmail.com</a>&gt; wrote:<br> <br> &gt; Please forgive my lack of technical skills-- I am not a developer.<br>
 <br> <br>Thank you for reporting this kernel bug!<br> <br><br> &gt; The bug report referenced below recently occurred on a pretty busy file<br> &gt; server of mine running Fedora 8 with a custom <a href="http://2.6.25.4">2.6.25.4</a> kernel.&nbsp;&nbsp;The attached<br>
 &gt; hardware RAID controller presents 12 LUNs, which were striped using the<br> &gt; Linux mdadm tools, and then the Linux LVM was used to create a single<br> &gt; logical volume.&nbsp;&nbsp;The filesystem used is XFS.<br> &gt;<br>
 &gt; Since upgrading from <a href="http://2.6.25.3">2.6.25.3</a> to <a href="http://2.6.25.4">2.6.25.4</a>, my log has shown a large number<br> &gt; of errors that I don&#39;t possess the skills to diagnose which component is<br>
 &gt; actually causing the errors.&nbsp;&nbsp;I can verify however, that the hardware is<br> &gt; good.&nbsp;&nbsp;I have run memtest86 and the vendor&#39;s diagnostic tools and verified<br> &gt; that everything is running well.<br> &gt;<br>
 &gt; The errors in /var/log/messages look like this:<br> &gt; May 19 06:30:30 ishtar kernel: Call Trace:<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;&lt;IRQ&gt;&nbsp;&nbsp;[&lt;ffffffff81075d35&gt;]<br> &gt; __alloc_pages+0x33f/0x35a<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff810934a4&gt;] new_slab+0x3f/0x242<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff81093bba&gt;]<br>
 &gt; __slab_alloc+0x212/0x43b<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128654c&gt;]<br> &gt; __netdev_alloc_skb+0x29/0x43<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128654c&gt;]<br> &gt; __netdev_alloc_skb+0x29/0x43<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff81094bd6&gt;]<br> &gt; __kmalloc_node_track_caller+0x75/0xab<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128584a&gt;] __alloc_skb+0x6a/0x133<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128654c&gt;]<br>
 &gt; __netdev_alloc_skb+0x29/0x43<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8123a405&gt;]<br> &gt; tg3_alloc_rx_skb+0xc4/0x153<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff81241343&gt;] tg3_poll+0x3be/0x7df<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8128c2ce&gt;]<br> &gt; net_rx_action+0xb6/0x1bf<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8103919b&gt;] __do_softirq+0x65/0xce<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100ce9c&gt;] call_softirq+0x1c/0x28<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100e544&gt;] do_softirq+0x2c/0x68<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff810390f2&gt;] irq_exit+0x3f/0x83<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100c221&gt;] ret_from_intr+0x0/0xa<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;&lt;EOI&gt;&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;]<br> &gt; default_idle+0x0/0x55<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8101c138&gt;]<br>
 &gt; lapic_next_event+0x0/0xa<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adf9&gt;] default_idle+0x31/0x55<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adf4&gt;] default_idle+0x2c/0x55<br>
 &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br> &gt; May 19 06:30:30 ishtar kernel:&nbsp;&nbsp;[&lt;ffffffff8100ae94&gt;] cpu_idle+0x77/0x9a<br> &gt;<br> &gt; The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor, 4GB<br>
 &gt; of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual<br> &gt; software RAID SATA disks for boot and swap.<br> &gt;<br> &gt; The call trace included with the bug report is the only one that appears in<br>
 &gt; the log with the &quot;cut here&quot; lines.<br> &gt;<br> <br> <br>OK, that is the TCP code performing self-checks and finding an<br> unexpected state.<br> <br> I was just a bit concerned about the &quot;segfault&quot; part of the<br>
 description.&nbsp;&nbsp;Because often when the kernel crashes this appears to the<br> operator as a userspace segfault.&nbsp;&nbsp;But the above trace isn&#39;t a kernel<br> crash - it&#39;s just a warning and the kernel does attept to fix things up<br>
 in this case.<br> <br> If however you were really seeing unexpected segfaults then it could be<br> the case that the kernel is crashing.&nbsp;&nbsp;But I assume this is not the<br> case, and that the text &quot;Oops&quot; does not appear in the logs.<br>
 </blockquote></div><br>
Comment 5 Ilpo Järvinen 2008-05-21 13:18:39 UTC 
On Wed, 21 May 2008, Brian Vowell wrote:

> Please forgive my lack of technical skills-- I am not a developer.

And I'm having problem in figuring out what is actually being reported 
here... so pleasy forgive my possibly stupid comments... :-)

> The bug report referenced below recently occurred on a pretty busy file
> server of mine running Fedora 8 with a custom 2.6.25.4 kernel.  The attached
> hardware RAID controller presents 12 LUNs, which were striped using the
> Linux mdadm tools, and then the Linux LVM was used to create a single
> logical volume.  The filesystem used is XFS.
> 
> Since upgrading from 2.6.25.3 to 2.6.25.4, my log has shown a large number
> of errors that I don't possess the skills to diagnose which component is
> actually causing the errors.  I can verify however, that the hardware is
> good.  I have run memtest86 and the vendor's diagnostic tools and verified
> that everything is running well.

At least the TCP thing is known to be in core TCP but it's just hard to 
debug and that's why it hasn't yet been closed, no hw is suspect due to 
it. But this other stacktrace seems to be something unrelated to TCP...:

> The errors in /var/log/messages look like this:
> May 19 06:30:30 ishtar kernel: Call Trace:
> May 19 06:30:30 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
> __alloc_pages+0x33f/0x35a
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> May 19 06:30:30 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
> May 19 06:30:30 ishtar kernel:  [<ffffffff81093bba>]
> __slab_alloc+0x212/0x43b
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff81094bd6>]
> __kmalloc_node_track_caller+0x75/0xab
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 19 06:30:30 ishtar kernel:  [<ffffffff8123a405>]
> tg3_alloc_rx_skb+0xc4/0x153
> May 19 06:30:30 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
> May 19 06:30:30 ishtar kernel:  [<ffffffff8128c2ce>]
> net_rx_action+0xb6/0x1bf
> May 19 06:30:30 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
> May 19 06:30:30 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
> May 19 06:30:30 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
> default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8101c138>]
> lapic_next_event+0x0/0xa
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
> May 19 06:30:30 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> 
> The hardware is an HP Proliant DL145G2, Dual Core Opteron 275 processor, 4GB
> of RAM, dual Broadcom GigE NICs bonded with Linux bonding driver, dual
> software RAID SATA disks for boot and swap.
> 
> The call trace included with the bug report is the only one that appears in
> the log with the "cut here" lines.
> 
> 
> 
> 
> On 5/21/08, Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> >
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> >
> > On Wed, 21 May 2008 02:45:59 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.orgwrote:
> >
> > > http://bugzilla.kernel.org/show_bug.cgi?id=10767
> > >
> > >            Summary: Seg Fault Instead of Swapping
> >
> > Strange description?
> >
> > >            Product: Networking
> > >            Version: 2.5
> > >      KernelVersion: 2.6.25.4
> > >           Platform: All
> > >         OS/Version: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: normal
> > >           Priority: P1
> > >          Component: IPV4
> > >         AssignedTo: shemminger@linux-foundation.org
> > >         ReportedBy: brian.vowell@gmail.com
> > >
> > >
> > > May 20 13:11:00 ishtar kernel: ------------[ cut here ]------------
> > > May 20 13:11:00 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:2539
> > > tcp_ack+0xd2b/0x191f()
> >
> > In 2.6.25 this is
> >
> >         if (WARN_ON(!tp->sacked_out && tp->fackets_out))
> >                 tp->fackets_out = 0;

...This TCP warning isn't sign of a problem that would cause corruption of 
any kind (and the state inconsistency gets repaired there as well!), we're 
debugging this already with couple of other people, so unless you have 
some very good reproducer case you'll just have to wait a bit.
Comment 6 Brian Vowell 2008-05-21 13:46:18 UTC 
On 5/21/08, Ilpo J
Comment 7 Ilpo Järvinen 2008-05-23 03:25:33 UTC 
On Wed, 21 May 2008, Brian Vowell wrote:

> On 5/21/08, Ilpo J
Comment 8 Brian Vowell 2008-05-23 09:07:08 UTC 
Here are some of the others that show in the logs:

This one repeats four times, and then again multiple times every minute
afterwards for about 30 minutes.
May 18 19:31:00 ishtar kernel: pdflush: page allocation failure. order:3,
mode:0x4020
May 18 19:31:00 ishtar kernel: Pid: 15300, comm: pdflush Not tainted
2.6.25.4 #1
May 18 19:31:00 ishtar kernel:
May 18 19:31:00 ishtar kernel: Call Trace:
May 18 19:31:00 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 18 19:31:00 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 18 19:31:00 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 18 19:31:00 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 18 19:31:00 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 18 19:31:00 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 18 19:31:00 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 18 19:31:00 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 18 19:31:00 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 18 19:31:00 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 18 19:31:00 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 18 19:31:00 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 18 19:31:00 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 18 19:31:00 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 18 19:31:00 ishtar kernel:  <EOI>  [<ffffffff8106fde3>]
find_get_pages_tag+0x7e/0x88
May 18 19:31:00 ishtar kernel:  [<ffffffff8107786a>]
pagevec_lookup_tag+0x1a/0x21
May 18 19:31:00 ishtar kernel:  [<ffffffff81076320>]
write_cache_pages+0x274/0x2d7
May 18 19:31:00 ishtar kernel:  [<ffffffff81075d85>] __writepage+0x0/0x23
May 18 19:31:00 ishtar kernel:  [<ffffffff810763bf>] do_writepages+0x20/0x2d
May 18 19:31:00 ishtar kernel:  [<ffffffff810b3804>]
__writeback_single_inode+0x140/0x26b
May 18 19:31:00 ishtar kernel:  [<ffffffff81048924>] ktime_get_ts+0x17/0x48
May 18 19:31:00 ishtar kernel:  [<ffffffff8811aaba>]
:dm_mod:dm_table_any_congested+0x46/0x6c
May 18 19:31:00 ishtar kernel:  [<ffffffff810b3cd1>]
sync_sb_inodes+0x1b5/0x28f
May 18 19:31:00 ishtar kernel:  [<ffffffff810b40ec>]
writeback_inodes+0x73/0xc4
May 18 19:31:00 ishtar kernel:  [<ffffffff81076b1b>]
background_writeout+0x90/0xca
May 18 19:31:00 ishtar kernel:  [<ffffffff81076fa6>] pdflush+0x0/0x1db
May 18 19:31:00 ishtar kernel:  [<ffffffff810770db>] pdflush+0x135/0x1db
May 18 19:31:00 ishtar kernel:  [<ffffffff81076a8b>]
background_writeout+0x0/0xca
May 18 19:31:00 ishtar kernel:  [<ffffffff81045d31>] kthread+0x47/0x76
May 18 19:31:00 ishtar kernel:  [<ffffffff8102fc15>] schedule_tail+0x28/0x5c
May 18 19:31:00 ishtar kernel:  [<ffffffff8100cb28>] child_rip+0xa/0x12
May 18 19:31:00 ishtar kernel:  [<ffffffff81045cea>] kthread+0x0/0x76
May 18 19:31:00 ishtar kernel:  [<ffffffff8100cb1e>] child_rip+0x0/0x12
May 18 19:31:00 ishtar kernel:
May 18 19:31:00 ishtar kernel: Mem-info:
May 18 19:31:00 ishtar kernel: Node 0 DMA per-cpu:
May 18 19:31:00 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 18 19:31:00 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 18 19:31:00 ishtar kernel: Node 0 DMA32 per-cpu:
May 18 19:31:00 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  40
May 18 19:31:00 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:  90
May 18 19:31:00 ishtar kernel: Node 0 Normal per-cpu:
May 18 19:31:00 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  76
May 18 19:31:00 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:  38
May 18 19:31:00 ishtar kernel: Active:118203 inactive:810010 dirty:63297
writeback:6742 unstable:0
May 18 19:31:00 ishtar kernel:  free:41617 slab:32326 mapped:111375
pagetables:1209 bounce:0
May 18 19:31:00 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? yes
May 18 19:31:00 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 18 19:31:00 ishtar kernel: Node 0 DMA32 free:153684kB min:6052kB
low:7564kB high:9076kB active:321388kB inactive:2431180kB present:3071328kB
pages_scanned:110 all_unreclaimable? no
May 18 19:31:00 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 18 19:31:00 ishtar kernel: Node 0 Normal free:2520kB min:2036kB
low:2544kB high:3052kB active:151424kB inactive:808860kB present:1034240kB
pages_scanned:0 all_unreclaimable? no
May 18 19:31:00 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 18 19:31:00 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 18 19:31:00 ishtar kernel: Node 0 DMA32: 3509*4kB 10039*8kB 3693*16kB
1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153724kB
May 18 19:31:00 ishtar kernel: Node 0 Normal: 173*4kB 212*8kB 2*16kB 1*32kB
0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2452kB
May 18 19:31:00 ishtar kernel: 918106 total pagecache pages
May 18 19:31:00 ishtar kernel: Swap cache: add 327, delete 327, find
2847/2867
May 18 19:31:00 ishtar kernel: Free swap  = 4192160kB
May 18 19:31:00 ishtar kernel: Total swap = 4192888kB
May 18 19:31:00 ishtar kernel: Free swap:       4192160kB
May 18 19:31:00 ishtar kernel: 1048576 pages of RAM
May 18 19:31:00 ishtar kernel: 33425 reserved pages
May 18 19:31:00 ishtar kernel: 718415 pages shared
May 18 19:31:00 ishtar kernel: 0 pages swap cached

Here's another later that night:
May 19 00:35:59 ishtar kernel: swapper: page allocation failure. order:3,
mode:0x4020
May 19 00:35:59 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 19 00:35:59 ishtar kernel:
May 19 00:35:59 ishtar kernel: Call Trace:
May 19 00:35:59 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 00:35:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 00:35:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 00:35:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:35:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:35:59 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 00:35:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 00:35:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:35:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 00:35:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 00:35:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 00:35:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 00:35:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 00:35:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 00:35:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 00:35:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 00:35:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 00:35:59 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 19 00:35:59 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 19 00:35:59 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 19 00:35:59 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 19 00:35:59 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 19 00:35:59 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 19 00:35:59 ishtar kernel:
May 19 00:35:59 ishtar kernel: Mem-info:
May 19 00:35:59 ishtar kernel: Node 0 DMA per-cpu:
May 19 00:35:59 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 19 00:35:59 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 19 00:35:59 ishtar kernel: Node 0 DMA32 per-cpu:
May 19 00:35:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:   0
May 19 00:35:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:   0
May 19 00:35:59 ishtar kernel: Node 0 Normal per-cpu:
May 19 00:35:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:   0
May 19 00:35:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:  30
May 19 00:35:59 ishtar kernel: Active:195037 inactive:769427 dirty:42097
writeback:345 unstable:0
May 19 00:35:59 ishtar kernel:  free:8758 slab:29079 mapped:164121
pagetables:1311 bounce:0
May 19 00:35:59 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? no
May 19 00:35:59 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 19 00:35:59 ishtar kernel: Node 0 DMA32 free:21912kB min:6052kB
low:7564kB high:9076kB active:524496kB inactive:2361884kB present:3071328kB
pages_scanned:0 all_unreclaimable? no
May 19 00:35:59 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 19 00:35:59 ishtar kernel: Node 0 Normal free:2856kB min:2036kB
low:2544kB high:3052kB active:255652kB inactive:715824kB present:1034240kB
pages_scanned:0 all_unreclaimable? no
May 19 00:35:59 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 19 00:35:59 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 19 00:35:59 ishtar kernel: Node 0 DMA32: 3397*4kB 932*8kB 37*16kB 1*32kB
0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 22052kB
May 19 00:35:59 ishtar kernel: Node 0 Normal: 417*4kB 142*8kB 2*16kB 0*32kB
1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2900kB
May 19 00:35:59 ishtar kernel: 950023 total pagecache pages
May 19 00:35:59 ishtar kernel: Swap cache: add 396, delete 396, find
4168/4198
May 19 00:35:59 ishtar kernel: Free swap  = 4192160kB
May 19 00:35:59 ishtar kernel: Total swap = 4192888kB
May 19 00:35:59 ishtar kernel: Free swap:       4192160kB
May 19 00:35:59 ishtar kernel: 1048576 pages of RAM
May 19 00:35:59 ishtar kernel: 33425 reserved pages
May 19 00:35:59 ishtar kernel: 869557 pages shared
May 19 00:35:59 ishtar kernel: 0 pages swap cached

and another:
May 19 00:40:59 ishtar kernel: xfs_fsr: page allocation failure. order:3,
mode:0x4020
May 19 00:40:59 ishtar kernel: Pid: 21473, comm: xfs_fsr Not tainted
2.6.25.4 #1
May 19 00:40:59 ishtar kernel:
May 19 00:40:59 ishtar kernel: Call Trace:
May 19 00:40:59 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 00:40:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 00:40:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 00:40:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:40:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:40:59 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 00:40:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 00:40:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 00:40:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 00:40:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 00:40:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 00:40:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 00:40:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 00:40:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 00:40:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 00:40:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 00:40:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 00:40:59 ishtar kernel:  <EOI>  [<ffffffff81072340>]
mempool_alloc+0x5c/0x110
May 19 00:40:59 ishtar kernel:  [<ffffffff8109415d>]
kmem_cache_alloc+0x47/0x65
May 19 00:40:59 ishtar kernel:  [<ffffffff81072340>]
mempool_alloc+0x5c/0x110
May 19 00:40:59 ishtar kernel:  [<ffffffff810bac2d>]
bio_alloc_bioset+0x89/0xd9
May 19 00:40:59 ishtar kernel:  [<ffffffff881190dc>]
:dm_mod:__split_bio+0xed/0x378
May 19 00:40:59 ishtar kernel:  [<ffffffff81151d0b>] xfs_iunlock+0x63/0x85
May 19 00:40:59 ishtar kernel:  [<ffffffff881198ac>]
:dm_mod:dm_request+0x104/0x126
May 19 00:40:59 ishtar kernel:  [<ffffffff811a2ec8>]
generic_make_request+0x31d/0x358
May 19 00:40:59 ishtar kernel:  [<ffffffff811a41d2>] submit_bio+0xdb/0xe2
May 19 00:40:59 ishtar kernel:  [<ffffffff810bacca>] bio_alloc+0x10/0x20
May 19 00:40:59 ishtar kernel:  [<ffffffff810bdade>]
mpage_bio_submit+0x22/0x26
May 19 00:40:59 ishtar kernel:  [<ffffffff810be492>]
do_mpage_readpage+0x406/0x498
May 19 00:40:59 ishtar kernel:  [<ffffffff8116f4f3>] xfs_get_blocks+0x0/0xe
May 19 00:40:59 ishtar kernel:  [<ffffffff8116f4f3>] xfs_get_blocks+0x0/0xe
May 19 00:40:59 ishtar kernel:  [<ffffffff81070343>]
add_to_page_cache+0x7c/0x90
May 19 00:40:59 ishtar kernel:  [<ffffffff8116f4f3>] xfs_get_blocks+0x0/0xe
May 19 00:40:59 ishtar kernel:  [<ffffffff810be6e4>]
mpage_readpages+0x98/0xd8
May 19 00:40:59 ishtar kernel:  [<ffffffff8116f4f3>] xfs_get_blocks+0x0/0xe
May 19 00:40:59 ishtar kernel:  [<ffffffff81075ab1>]
__alloc_pages+0xbb/0x35a
May 19 00:40:59 ishtar kernel:  [<ffffffff8107734c>]
__do_page_cache_readahead+0xec/0x164
May 19 00:40:59 ishtar kernel:  [<ffffffff8107762e>]
ondemand_readahead+0x14c/0x15b
May 19 00:40:59 ishtar kernel:  [<ffffffff81071a68>]
generic_file_aio_read+0x1e9/0x4f4
May 19 00:40:59 ishtar kernel:  [<ffffffff811758c8>] xfs_read+0x185/0x1f7
May 19 00:40:59 ishtar kernel:  [<ffffffff81099418>] do_sync_read+0xd1/0x118
May 19 00:40:59 ishtar kernel:  [<ffffffff81173fd7>] xfs_ioctl+0x361/0x622
May 19 00:40:59 ishtar kernel:  [<ffffffff81045e65>]
autoremove_wake_function+0x0/0x2e
May 19 00:40:59 ishtar kernel:  [<ffffffff81099c0d>] vfs_read+0xaa/0x152
May 19 00:40:59 ishtar kernel:  [<ffffffff81099fe6>] sys_read+0x45/0x6e
May 19 00:40:59 ishtar kernel:  [<ffffffff8100bd84>] sysret_signal+0x1c/0x27
May 19 00:40:59 ishtar kernel:  [<ffffffff8100bcfb>]
system_call_after_swapgs+0x7b/0x80
May 19 00:40:59 ishtar kernel:
May 19 00:40:59 ishtar kernel: Mem-info:
May 19 00:40:59 ishtar kernel: Node 0 DMA per-cpu:
May 19 00:40:59 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 19 00:40:59 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 19 00:40:59 ishtar kernel: Node 0 DMA32 per-cpu:
May 19 00:40:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:   0
May 19 00:40:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:   0
May 19 00:40:59 ishtar kernel: Node 0 Normal per-cpu:
May 19 00:40:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:   0
May 19 00:40:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:  30
May 19 00:40:59 ishtar kernel: Active:181485 inactive:782420 dirty:44402
writeback:287 unstable:0
May 19 00:40:59 ishtar kernel:  free:9403 slab:28952 mapped:156908
pagetables:1303 bounce:0
May 19 00:40:59 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? no
May 19 00:40:59 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 19 00:40:59 ishtar kernel: Node 0 DMA32 free:24732kB min:6052kB
low:7564kB high:9076kB active:480508kB inactive:2403044kB present:3071328kB
pages_scanned:0 all_unreclaimable? no
May 19 00:40:59 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 19 00:40:59 ishtar kernel: Node 0 Normal free:2616kB min:2036kB
low:2544kB high:3052kB active:245432kB inactive:726636kB present:1034240kB
pages_scanned:0 all_unreclaimable? no
May 19 00:40:59 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 19 00:40:59 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 19 00:40:59 ishtar kernel: Node 0 DMA32: 3161*4kB 1316*8kB 83*16kB
1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24788kB
May 19 00:40:59 ishtar kernel: Node 0 Normal: 290*4kB 156*8kB 9*16kB 1*32kB
1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2648kB
May 19 00:40:59 ishtar kernel: 949523 total pagecache pages
May 19 00:40:59 ishtar kernel: Swap cache: add 396, delete 396, find
4168/4198
May 19 00:40:59 ishtar kernel: Free swap  = 4192160kB
May 19 00:40:59 ishtar kernel: Total swap = 4192888kB
May 19 00:40:59 ishtar kernel: Free swap:       4192160kB

and this one:
May 19 01:05:59 ishtar kernel: rsync: page allocation failure. order:3,
mode:0x4020
May 19 01:05:59 ishtar kernel: Pid: 21078, comm: rsync Not tainted 2.6.25.4#1
May 19 01:05:59 ishtar kernel:
May 19 01:05:59 ishtar kernel: Call Trace:
May 19 01:05:59 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 01:05:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 01:05:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 01:05:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:05:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:05:59 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 01:05:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 01:05:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:05:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 01:05:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 01:05:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 01:05:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 01:05:59 ishtar kernel:  [<ffffffff81074094>]
drain_local_pages+0x0/0xd
May 19 01:05:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 01:05:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 01:05:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 01:05:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:05:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 01:05:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:05:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 01:05:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 01:05:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 01:05:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 01:05:59 ishtar kernel:  [<ffffffff81074094>]
drain_local_pages+0x0/0xd
May 19 01:05:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 01:05:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 01:05:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 01:05:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:05:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 01:05:59 ishtar kernel:  <EOI>  [<ffffffff8101b62f>]
__smp_call_function_mask+0xa5/0xc5
May 19 01:05:59 ishtar kernel:  [<ffffffff8101b61a>]
__smp_call_function_mask+0x90/0xc5
May 19 01:05:59 ishtar kernel:  [<ffffffff81074094>]
drain_local_pages+0x0/0xd
May 19 01:05:59 ishtar kernel:  [<ffffffff81074094>]
drain_local_pages+0x0/0xd
May 19 01:05:59 ishtar kernel:  [<ffffffff8101b698>]
smp_call_function_mask+0x49/0x57
May 19 01:05:59 ishtar kernel:  [<ffffffff81078fbb>]
isolate_pages_global+0x0/0x2f
May 19 01:05:59 ishtar kernel:  [<ffffffff81074094>]
drain_local_pages+0x0/0xd
May 19 01:05:59 ishtar kernel:  [<ffffffff81038d2f>] on_each_cpu+0x10/0x22
May 19 01:05:59 ishtar kernel:  [<ffffffff81075c2d>]
__alloc_pages+0x237/0x35a
May 19 01:05:59 ishtar kernel:  [<ffffffff810933d8>]
kmalloc_large_node+0xc2/0x101
May 19 01:05:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 01:05:59 ishtar kernel:  [<ffffffff81281b78>]
sock_alloc_send_skb+0x93/0x1e1
May 19 01:05:59 ishtar kernel:  [<ffffffff812f9972>] io_schedule+0x7c/0x9f
May 19 01:05:59 ishtar kernel:  [<ffffffff812e4644>]
unix_stream_sendmsg+0x18c/0x355
May 19 01:05:59 ishtar kernel:  [<ffffffff8127f170>]
sock_aio_write+0xf7/0x106
May 19 01:05:59 ishtar kernel:  [<ffffffff81099300>]
do_sync_write+0xd1/0x118
May 19 01:05:59 ishtar kernel:  [<ffffffff8104800f>]
enqueue_hrtimer+0xd1/0xde
May 19 01:05:59 ishtar kernel:  [<ffffffff81045e65>]
autoremove_wake_function+0x0/0x2e
May 19 01:05:59 ishtar kernel:  [<ffffffff812e29e0>] unix_poll+0x0/0x96
May 19 01:05:59 ishtar kernel:  [<ffffffff81099acd>] vfs_write+0xc0/0x156
May 19 01:05:59 ishtar kernel:  [<ffffffff8109a054>] sys_write+0x45/0x6e
May 19 01:05:59 ishtar kernel:  [<ffffffff8100bcfb>]
system_call_after_swapgs+0x7b/0x80
May 19 01:05:59 ishtar kernel:
May 19 01:05:59 ishtar kernel: Mem-info:
May 19 01:05:59 ishtar kernel: Node 0 DMA per-cpu:
May 19 01:05:59 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 19 01:05:59 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 19 01:05:59 ishtar kernel: Node 0 DMA32 per-cpu:
May 19 01:05:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd: 117
May 19 01:05:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:   0
May 19 01:05:59 ishtar kernel: Node 0 Normal per-cpu:
May 19 01:05:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  64
May 19 01:05:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd:  72
May 19 01:05:59 ishtar kernel: Active:107995 inactive:858970 dirty:44658
writeback:256 unstable:0
May 19 01:05:59 ishtar kernel:  free:6582 slab:28806 mapped:84981
pagetables:1141 bounce:0
May 19 01:05:59 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? no
May 19 01:05:59 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 19 01:05:59 ishtar kernel: Node 0 DMA32 free:12980kB min:6052kB
low:7564kB high:9076kB active:284820kB inactive:2611572kB present:3071328kB
pages_scanned:97 all_unreclaimable? no
May 19 01:05:59 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 19 01:05:59 ishtar kernel: Node 0 Normal free:3084kB min:2036kB
low:2544kB high:3052kB active:147160kB inactive:824308kB present:1034240kB
pages_scanned:40 all_unreclaimable? no
May 19 01:05:59 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 19 01:05:59 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 19 01:05:59 ishtar kernel: Node 0 DMA32: 2178*4kB 382*8kB 57*16kB 0*32kB
1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13000kB
May 19 01:05:59 ishtar kernel: Node 0 Normal: 542*4kB 91*8kB 1*16kB 1*32kB
1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3008kB
May 19 01:05:59 ishtar kernel: 952545 total pagecache pages
May 19 01:05:59 ishtar kernel: Swap cache: add 396, delete 396, find
4168/4198
May 19 01:05:59 ishtar kernel: Free swap  = 4192160kB
May 19 01:05:59 ishtar kernel: Total swap = 4192888kB
May 19 01:05:59 ishtar kernel: Free swap:       4192160kB
May 19 01:05:59 ishtar kernel: 1048576 pages of RAM
May 19 01:05:59 ishtar kernel: 33425 reserved pages
May 19 01:05:59 ishtar kernel: 692754 pages shared
May 19 01:05:59 ishtar kernel: 0 pages swap cached

May 19 01:23:59 ishtar kernel: md3_raid5: page allocation failure. order:3,
mode:0x4020
May 19 01:23:59 ishtar kernel: Pid: 1557, comm: md3_raid5 Not tainted
2.6.25.4 #1
May 19 01:23:59 ishtar kernel:
May 19 01:23:59 ishtar kernel: Call Trace:
May 19 01:23:59 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:23:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 01:23:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 01:23:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 01:23:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 01:23:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 01:23:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 01:23:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 01:23:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:23:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:23:59 ishtar kernel:  [<ffffffff8803244d>]
:raid456:ops_complete_prexor+0x0/0x7
May 19 01:23:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 01:23:59 ishtar kernel:  <EOI>  [<ffffffff88025000>]
:xor:xor_sse_2+0x0/0x1e2
May 19 01:23:59 ishtar kernel:  [<ffffffff8802508d>]
:xor:xor_sse_2+0x8d/0x1e2
May 19 01:23:59 ishtar kernel:  [<ffffffff8802d369>]
:async_xor:async_xor+0x369/0x3fb
May 19 01:23:59 ishtar kernel:  [<ffffffff8803250b>]
:raid456:ops_run_prexor+0xb7/0xc0
May 19 01:23:59 ishtar kernel:  [<ffffffff8803244d>]
:raid456:ops_complete_prexor+0x0/0x7
May 19 01:23:59 ishtar kernel:  [<ffffffff88032c06>]
:raid456:raid5_run_ops+0x157/0x4ce
May 19 01:23:59 ishtar kernel:  [<ffffffff88033a2b>]
:raid456:handle_stripe5+0xaae/0xac5
May 19 01:23:59 ishtar kernel:  [<ffffffff88035bad>]
:raid456:handle_stripe+0xd2f/0xd75
May 19 01:23:59 ishtar kernel:  [<ffffffff8102faf4>] hrtick_set+0x88/0xf6
May 19 01:23:59 ishtar kernel:  [<ffffffff812f9766>] thread_return+0x69/0xad
May 19 01:23:59 ishtar kernel:  [<ffffffff8102a7ca>] __wake_up+0x38/0x4e
May 19 01:23:59 ishtar kernel:  [<ffffffff88035f01>]
:raid456:raid5d+0x30e/0x325
May 19 01:23:59 ishtar kernel:  [<ffffffff81269898>] md_thread+0xde/0xfa
May 19 01:23:59 ishtar kernel:  [<ffffffff81045e65>]
autoremove_wake_function+0x0/0x2e
May 19 01:23:59 ishtar kernel:  [<ffffffff812697ba>] md_thread+0x0/0xfa
May 19 01:23:59 ishtar kernel:  [<ffffffff81045d31>] kthread+0x47/0x76
May 19 01:23:59 ishtar kernel:  [<ffffffff8102fc15>] schedule_tail+0x28/0x5c
May 19 01:23:59 ishtar kernel:  [<ffffffff8100cb28>] child_rip+0xa/0x12
May 19 01:23:59 ishtar kernel:  [<ffffffff81045cea>] kthread+0x0/0x76
May 19 01:23:59 ishtar kernel:  [<ffffffff8100cb1e>] child_rip+0x0/0x12
May 19 01:23:59 ishtar kernel:
May 19 01:23:59 ishtar kernel: Mem-info:
May 19 01:23:59 ishtar kernel: Node 0 DMA per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 19 01:23:59 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 19 01:23:59 ishtar kernel: Node 0 DMA32 per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  53
May 19 01:23:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd: 178
May 19 01:23:59 ishtar kernel: Node 0 Normal per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  30
May 19 01:23:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd: 169
May 19 01:23:59 ishtar kernel: Active:112521 inactive:850897 dirty:39689
writeback:269 unstable:0
May 19 01:23:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd: 169
May 19 01:23:59 ishtar kernel: Active:112521 inactive:850897 dirty:39689
writeback:269 unstable:0
May 19 01:23:59 ishtar kernel:  free:9835 slab:28797 mapped:105998
pagetables:1165 bounce:0
May 19 01:23:59 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? yes
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 19 01:23:59 ishtar kernel: Node 0 DMA32 free:26628kB min:6052kB
low:7564kB high:9076kB active:287312kB inactive:2594200kB present:3071328kB
pages_scanned:0 all_unreclaimable? no
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 19 01:23:59 ishtar kernel: Node 0 Normal free:2448kB min:2036kB
low:2544kB high:3052kB active:162772kB inactive:809388kB present:1034240kB
pages_scanned:0 all_unreclaimable? no
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 19 01:23:59 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 19 01:23:59 ishtar kernel: Node 0 DMA32: 2457*4kB 1830*8kB 121*16kB
0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 26596kB
May 19 01:23:59 ishtar kernel: Node 0 Normal: 103*4kB 236*8kB 3*16kB 0*32kB
1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2412kB
May 19 01:23:59 ishtar kernel: 949319 total pagecache pages
May 19 01:23:59 ishtar kernel: Swap cache: add 396, delete 396, find
4168/4198
May 19 01:23:59 ishtar kernel: Free swap  = 4192160kB
May 19 01:23:59 ishtar kernel: Total swap = 4192888kB
May 19 01:23:59 ishtar kernel: Free swap:       4192160kB

May 19 01:23:59 ishtar kernel: rklogd: page allocation failure. order:3,
mode:0x4020
May 19 01:23:59 ishtar kernel: Pid: 1937, comm: rklogd Not tainted 2.6.25.4#1
May 19 01:23:59 ishtar kernel:
May 19 01:23:59 ishtar kernel: Call Trace:
May 19 01:23:59 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
__alloc_pages+0x33f/0x35a
May 19 01:23:59 ishtar kernel:  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
May 19 01:23:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 01:23:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff81094bd6>]
__kmalloc_node_track_caller+0x75/0xab
May 19 01:23:59 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
May 19 01:23:59 ishtar kernel:  [<ffffffff8128654c>]
__netdev_alloc_skb+0x29/0x43
May 19 01:23:59 ishtar kernel:  [<ffffffff8123a405>]
tg3_alloc_rx_skb+0xc4/0x153
May 19 01:23:59 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df
May 19 01:23:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 19 01:23:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 19 01:23:59 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 19 01:23:59 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 19 01:23:59 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 19 01:23:59 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 19 01:23:59 ishtar kernel:  <EOI>  [<ffffffff8117015f>]
xfs_vm_releasepage+0x0/0xa4
May 19 01:23:59 ishtar kernel:  [<ffffffff8107919d>]
remove_mapping+0xa6/0xbd
May 19 01:23:59 ishtar kernel:  [<ffffffff8107918e>]
remove_mapping+0x97/0xbd
May 19 01:23:59 ishtar kernel:  [<ffffffff8107962f>]
shrink_page_list+0x47b/0x548
May 19 01:23:59 ishtar kernel:  [<ffffffff81078f36>]
isolate_lru_pages+0x14c/0x1d1
May 19 01:23:59 ishtar kernel:  [<ffffffff81079859>]
shrink_inactive_list+0x15d/0x378
May 19 01:23:59 ishtar kernel:  [<ffffffff81079b56>] shrink_zone+0xe2/0x105
May 19 01:23:59 ishtar kernel:  [<ffffffff81079e5e>]
try_to_free_pages+0x18b/0x297
May 19 01:23:59 ishtar kernel:  [<ffffffff81078fbb>]
isolate_pages_global+0x0/0x2f
May 19 01:23:59 ishtar kernel:  [<ffffffff81075c08>]
__alloc_pages+0x212/0x35a
May 19 01:23:59 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
May 19 01:23:59 ishtar kernel:  [<ffffffff81093bba>]
__slab_alloc+0x212/0x43b
May 19 01:23:59 ishtar kernel:  [<ffffffff8117467d>]
xfs_vn_follow_link+0x27/0x72
May 19 01:23:59 ishtar kernel:  [<ffffffff8109414b>]
kmem_cache_alloc+0x35/0x65
May 19 01:23:59 ishtar kernel:  [<ffffffff8117467d>]
xfs_vn_follow_link+0x27/0x72
May 19 01:23:59 ishtar kernel:  [<ffffffff810a2527>]
__link_path_walk+0x9f9/0xd62
May 19 01:23:59 ishtar kernel:  [<ffffffff8102fb0d>] hrtick_set+0xa1/0xf6
May 19 01:23:59 ishtar kernel:  [<ffffffff810a28ed>] path_walk+0x5d/0xb8
May 19 01:23:59 ishtar kernel:  [<ffffffff810a2bb0>]
do_path_lookup+0x154/0x1ce
May 19 01:23:59 ishtar kernel:  [<ffffffff810a18f5>] getname+0x140/0x1a4
May 19 01:23:59 ishtar kernel:  [<ffffffff810a33ee>]
__user_walk_fd+0x37/0x4c
May 19 01:23:59 ishtar kernel:  [<ffffffff8109c613>] vfs_stat_fd+0x1b/0x4a
May 19 01:23:59 ishtar kernel:  [<ffffffff81045e65>]
autoremove_wake_function+0x0/0x2e
May 19 01:23:59 ishtar kernel:  [<ffffffff810db690>] kmsg_read+0x3a/0x44
May 19 01:23:59 ishtar kernel:  [<ffffffff8109c69e>] sys_newstat+0x19/0x31
May 19 01:23:59 ishtar kernel:  [<ffffffff8104a1bb>]
getnstimeofday+0x2f/0x83
May 19 01:23:59 ishtar kernel:  [<ffffffff8104a21f>]
do_gettimeofday+0x10/0x32
May 19 01:23:59 ishtar kernel:  [<ffffffff81038c3f>]
sys_gettimeofday+0x2b/0x62
May 19 01:23:59 ishtar kernel:  [<ffffffff8100bcfb>]
system_call_after_swapgs+0x7b/0x80
May 19 01:23:59 ishtar kernel:
May 19 01:23:59 ishtar kernel: Mem-info:
May 19 01:23:59 ishtar kernel: Node 0 DMA per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 19 01:23:59 ishtar kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 19 01:23:59 ishtar kernel: Node 0 DMA32 per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  53
May 19 01:23:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd: 146
May 19 01:23:59 ishtar kernel: Node 0 Normal per-cpu:
May 19 01:23:59 ishtar kernel: CPU    0: hi:  186, btch:  31 usd:  30
May 19 01:23:59 ishtar kernel: CPU    1: hi:  186, btch:  31 usd: 183
May 19 01:23:59 ishtar kernel: Active:112521 inactive:850902 dirty:39689
writeback:269 unstable:0
May 19 01:23:59 ishtar kernel:  free:9835 slab:28797 mapped:105998
pagetables:1165 bounce:0
May 19 01:23:59 ishtar kernel: Node 0 DMA free:10264kB min:16kB low:20kB
high:24kB active:0kB inactive:0kB present:10080kB pages_scanned:0
all_unreclaimable? yes
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 2999 4009 4009
May 19 01:23:59 ishtar kernel: Node 0 DMA32 free:26628kB min:6052kB
low:7564kB high:9076kB active:287312kB inactive:2594348kB present:3071328kB
pages_scanned:0 all_unreclaimable? no
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 0 1010 1010
May 19 01:23:59 ishtar kernel: Node 0 Normal free:2448kB min:2036kB
low:2544kB high:3052kB active:162772kB inactive:809260kB present:1034240kB
pages_scanned:32 all_unreclaimable? no
May 19 01:23:59 ishtar kernel: lowmem_reserve[]: 0 0 0 0
May 19 01:23:59 ishtar kernel: Node 0 DMA: 4*4kB 3*8kB 1*16kB 3*32kB 2*64kB
2*128kB 2*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10264kB
May 19 01:23:59 ishtar kernel: Node 0 DMA32: 2457*4kB 1830*8kB 119*16kB
0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 26564kB
May 19 01:23:59 ishtar kernel: Node 0 Normal: 103*4kB 236*8kB 3*16kB 0*32kB
1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2412kB
May 19 01:23:59 ishtar kernel: 949331 total pagecache pages
May 19 01:23:59 ishtar kernel: Swap cache: add 396, delete 396, find
4168/4198
May 19 01:23:59 ishtar kernel: Free swap  = 4192160kB
May 19 01:23:59 ishtar kernel: Total swap = 4192888kB
May 19 01:23:59 ishtar kernel: Free swap:       4192160kB
May 19 01:23:59 ishtar kernel: 1048576 pages of RAM
May 19 01:23:59 ishtar kernel: 33425 reserved pages
May 19 01:23:59 ishtar kernel: 717421 pages shared
May 19 01:23:59 ishtar kernel: 0 pages swap cached




On 5/23/08, Ilpo J
Comment 9 Brian Vowell 2008-05-23 10:24:26 UTC 
I applied the ipv4 patch.  Here are two traces that just showed up:

------------[ cut here ]------------
WARNING: at net/ipv4/tcp_input.c:3297 tcp_ack+0xd58/0xeba()
Modules linked in: dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util
button jedec_probe cfi_probe gen_probe ck804xrom i2c_nforce2 mtd chipreg
map_funcs k8temp sg hwmon pcspkr i2c_core serio_raw pata_amd cciss
ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor
async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last
unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Not tainted 2.6.25.4 #1

Call Trace:
 <IRQ>  [<ffffffff81034481>] warn_on_slowpath+0x51/0x63
 [<ffffffff8128cdaf>] dev_queue_xmit+0x25b/0x284
 [<ffffffff812ab9fa>] ip_queue_xmit+0x2cc/0x31f
 [<ffffffff812b0e3b>] sk_stream_alloc_skb+0x2f/0xd2
 [<ffffffff8104a1bb>] getnstimeofday+0x2f/0x83
 [<ffffffff812bb83d>] tcp_transmit_skb+0x775/0x7a4
 [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
 [<ffffffff812ba74b>] tcp_rcv_established+0x7b9/0x8ce
 [<ffffffff812c05cd>] tcp_v4_do_rcv+0x2c2/0x48b
 [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
 [<ffffffff812ae376>] __inet_lookup_established+0xdf/0x17b
 [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
 [<ffffffff812a7558>] ip_local_deliver+0xd4/0x18e
 [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
 [<ffffffff8128a2b7>] netif_receive_skb+0x351/0x372
 [<ffffffff8124150d>] tg3_poll+0x588/0x7df
 [<ffffffff8128c2ce>] net_rx_action+0xb6/0x1bf
 [<ffffffff8103919b>] __do_softirq+0x65/0xce
 [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
 [<ffffffff8100e544>] do_softirq+0x2c/0x68
 [<ffffffff810390f2>] irq_exit+0x3f/0x83
 [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
 [<ffffffff8100c221>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8101c138>] lapic_next_event+0x0/0xa
 [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8100adf9>] default_idle+0x31/0x55
 [<ffffffff8100adf4>] default_idle+0x2c/0x55
 [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8100ae94>] cpu_idle+0x77/0x9a

---[ end trace aae73dd976dfcd54 ]---
TCP wq(s)  S                <
TCP wq(h) ++h-----+-----+---<
l0 s1 f2 p18 seq: wq1288576867, su1288576867 hs1288579763 sn1288602883


------------[ cut here ]------------
WARNING: at net/ipv4/tcp_input.c:3297 tcp_ack+0xd58/0xeba()
Modules linked in: dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util
button jedec_probe cfi_probe gen_probe ck804xrom i2c_nforce2 mtd chipreg
map_funcs k8temp sg hwmon pcspkr i2c_core serio_raw pata_amd cciss
ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor
async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last
unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Not tainted 2.6.25.4 #1

Call Trace:
 <IRQ>  [<ffffffff81034481>] warn_on_slowpath+0x51/0x63
 [<ffffffff8122d718>] bond_dev_queue_xmit+0x18a/0x191
 [<ffffffff8122e21c>] bond_xmit_roundrobin+0x9d/0xb3
 [<ffffffff8102ba0d>] task_rq_lock+0x3d/0x73
 [<ffffffff8102be0b>] try_to_wake_up+0x1ae/0x1bf
 [<ffffffff8128cdaf>] dev_queue_xmit+0x25b/0x284
 [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
 [<ffffffff8102a776>] __wake_up_sync+0x3a/0x56
 [<ffffffff812ba74b>] tcp_rcv_established+0x7b9/0x8ce
 [<ffffffff812c05cd>] tcp_v4_do_rcv+0x2c2/0x48b
 [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
 [<ffffffff812ae376>] __inet_lookup_established+0xdf/0x17b
 [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
 [<ffffffff812a7558>] ip_local_deliver+0xd4/0x18e
 [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
 [<ffffffff8128a2b7>] netif_receive_skb+0x351/0x372
 [<ffffffff8124150d>] tg3_poll+0x588/0x7df
 [<ffffffff8128c2ce>] net_rx_action+0xb6/0x1bf
 [<ffffffff8103919b>] __do_softirq+0x65/0xce
 [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
 [<ffffffff8100e544>] do_softirq+0x2c/0x68
 [<ffffffff810390f2>] irq_exit+0x3f/0x83
 [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
 [<ffffffff8100c221>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8101c138>] lapic_next_event+0x0/0xa
 [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8100adf9>] default_idle+0x31/0x55
 [<ffffffff8100adf4>] default_idle+0x2c/0x55
 [<ffffffff8100adc8>] default_idle+0x0/0x55
 [<ffffffff8100ae94>] cpu_idle+0x77/0x9a

---[ end trace aae73dd976dfcd54 ]---
TCP wq(s)  S                <
TCP wq(h) ++h-----+-----+---<
l0 s1 f2 p18 seq: wq1288576867, su1288576867 hs1288579763 sn1288602883
I applied the ipv4 patch.&nbsp; Here are two traces that just showed up:<br><br>------------[ cut here ]------------<br>WARNING: at net/ipv4/tcp_input.c:3297 tcp_ack+0xd58/0xeba()<br>Modules linked in: dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util button jedec_probe cfi_probe gen_probe ck804xrom i2c_nforce2 mtd chipreg map_funcs k8temp sg hwmon pcspkr i2c_core serio_raw pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last unloaded: scsi_wait_scan]<br>
Pid: 0, comm: swapper Not tainted <a href="http://2.6.25.4">2.6.25.4</a> #1<br><br>Call Trace:<br>&nbsp;&lt;IRQ&gt;&nbsp; [&lt;ffffffff81034481&gt;] warn_on_slowpath+0x51/0x63<br>&nbsp;[&lt;ffffffff8128cdaf&gt;] dev_queue_xmit+0x25b/0x284<br>
&nbsp;[&lt;ffffffff812ab9fa&gt;] ip_queue_xmit+0x2cc/0x31f<br>&nbsp;[&lt;ffffffff812b0e3b&gt;] sk_stream_alloc_skb+0x2f/0xd2<br>&nbsp;[&lt;ffffffff8104a1bb&gt;] getnstimeofday+0x2f/0x83<br>&nbsp;[&lt;ffffffff812bb83d&gt;] tcp_transmit_skb+0x775/0x7a4<br>
&nbsp;[&lt;ffffffff812b760a&gt;] tcp_ack+0xd58/0xeba<br>&nbsp;[&lt;ffffffff812ba74b&gt;] tcp_rcv_established+0x7b9/0x8ce<br>&nbsp;[&lt;ffffffff812c05cd&gt;] tcp_v4_do_rcv+0x2c2/0x48b<br>&nbsp;[&lt;ffffffff8129a07a&gt;] __qdisc_run+0xf6/0x1c8<br>
&nbsp;[&lt;ffffffff812ae376&gt;] __inet_lookup_established+0xdf/0x17b<br>&nbsp;[&lt;ffffffff812c2528&gt;] tcp_v4_rcv+0x6a3/0x700<br>&nbsp;[&lt;ffffffff812a7558&gt;] ip_local_deliver+0xd4/0x18e<br>&nbsp;[&lt;ffffffff812a7b0d&gt;] ip_rcv+0x4fb/0x53a<br>
&nbsp;[&lt;ffffffff8128a2b7&gt;] netif_receive_skb+0x351/0x372<br>&nbsp;[&lt;ffffffff8124150d&gt;] tg3_poll+0x588/0x7df<br>&nbsp;[&lt;ffffffff8128c2ce&gt;] net_rx_action+0xb6/0x1bf<br>&nbsp;[&lt;ffffffff8103919b&gt;] __do_softirq+0x65/0xce<br>
&nbsp;[&lt;ffffffff8100ce9c&gt;] call_softirq+0x1c/0x28<br>&nbsp;[&lt;ffffffff8100e544&gt;] do_softirq+0x2c/0x68<br>&nbsp;[&lt;ffffffff810390f2&gt;] irq_exit+0x3f/0x83<br>&nbsp;[&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br>&nbsp;[&lt;ffffffff8100c221&gt;] ret_from_intr+0x0/0xa<br>
&nbsp;&lt;EOI&gt;&nbsp; [&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8101c138&gt;] lapic_next_event+0x0/0xa<br>&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8100adf9&gt;] default_idle+0x31/0x55<br>
&nbsp;[&lt;ffffffff8100adf4&gt;] default_idle+0x2c/0x55<br>&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8100ae94&gt;] cpu_idle+0x77/0x9a<br><br>---[ end trace aae73dd976dfcd54 ]---<br>TCP wq(s)&nbsp; S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;<br>
TCP wq(h) ++h-----+-----+---&lt;<br>l0 s1 f2 p18 seq: wq1288576867, su1288576867 hs1288579763 sn1288602883<br><br><br>------------[ cut here ]------------<br>WARNING: at net/ipv4/tcp_input.c:3297 tcp_ack+0xd58/0xeba()<br>
Modules linked in: dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util button jedec_probe cfi_probe gen_probe ck804xrom i2c_nforce2 mtd chipreg map_funcs k8temp sg hwmon pcspkr i2c_core serio_raw pata_amd cciss ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last unloaded: scsi_wait_scan]<br>
Pid: 0, comm: swapper Not tainted <a href="http://2.6.25.4">2.6.25.4</a> #1<br><br>Call Trace:<br>&nbsp;&lt;IRQ&gt;&nbsp; [&lt;ffffffff81034481&gt;] warn_on_slowpath+0x51/0x63<br>&nbsp;[&lt;ffffffff8122d718&gt;] bond_dev_queue_xmit+0x18a/0x191<br>
&nbsp;[&lt;ffffffff8122e21c&gt;] bond_xmit_roundrobin+0x9d/0xb3<br>&nbsp;[&lt;ffffffff8102ba0d&gt;] task_rq_lock+0x3d/0x73<br>&nbsp;[&lt;ffffffff8102be0b&gt;] try_to_wake_up+0x1ae/0x1bf<br>&nbsp;[&lt;ffffffff8128cdaf&gt;] dev_queue_xmit+0x25b/0x284<br>
&nbsp;[&lt;ffffffff812b760a&gt;] tcp_ack+0xd58/0xeba<br>&nbsp;[&lt;ffffffff8102a776&gt;] __wake_up_sync+0x3a/0x56<br>&nbsp;[&lt;ffffffff812ba74b&gt;] tcp_rcv_established+0x7b9/0x8ce<br>&nbsp;[&lt;ffffffff812c05cd&gt;] tcp_v4_do_rcv+0x2c2/0x48b<br>
&nbsp;[&lt;ffffffff8129a07a&gt;] __qdisc_run+0xf6/0x1c8<br>&nbsp;[&lt;ffffffff812ae376&gt;] __inet_lookup_established+0xdf/0x17b<br>&nbsp;[&lt;ffffffff812c2528&gt;] tcp_v4_rcv+0x6a3/0x700<br>&nbsp;[&lt;ffffffff812a7558&gt;] ip_local_deliver+0xd4/0x18e<br>
&nbsp;[&lt;ffffffff812a7b0d&gt;] ip_rcv+0x4fb/0x53a<br>&nbsp;[&lt;ffffffff8128a2b7&gt;] netif_receive_skb+0x351/0x372<br>&nbsp;[&lt;ffffffff8124150d&gt;] tg3_poll+0x588/0x7df<br>&nbsp;[&lt;ffffffff8128c2ce&gt;] net_rx_action+0xb6/0x1bf<br>&nbsp;[&lt;ffffffff8103919b&gt;] __do_softirq+0x65/0xce<br>
&nbsp;[&lt;ffffffff8100ce9c&gt;] call_softirq+0x1c/0x28<br>&nbsp;[&lt;ffffffff8100e544&gt;] do_softirq+0x2c/0x68<br>&nbsp;[&lt;ffffffff810390f2&gt;] irq_exit+0x3f/0x83<br>&nbsp;[&lt;ffffffff8100e813&gt;] do_IRQ+0x13e/0x15f<br>&nbsp;[&lt;ffffffff8100c221&gt;] ret_from_intr+0x0/0xa<br>
&nbsp;&lt;EOI&gt;&nbsp; [&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8101c138&gt;] lapic_next_event+0x0/0xa<br>&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8100adf9&gt;] default_idle+0x31/0x55<br>
&nbsp;[&lt;ffffffff8100adf4&gt;] default_idle+0x2c/0x55<br>&nbsp;[&lt;ffffffff8100adc8&gt;] default_idle+0x0/0x55<br>&nbsp;[&lt;ffffffff8100ae94&gt;] cpu_idle+0x77/0x9a<br><br>---[ end trace aae73dd976dfcd54 ]---<br>TCP wq(s)&nbsp; S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;<br>
TCP wq(h) ++h-----+-----+---&lt;<br>l0 s1 f2 p18 seq: wq1288576867, su1288576867 hs1288579763 sn1288602883<br><br>
Comment 10 Ilpo Järvinen 2008-05-23 11:02:02 UTC 
On Fri, 23 May 2008, Brian Vowell wrote:

> I applied the ipv4 patch.  Here are two traces that just showed up:
> 
> ------------[ cut here ]------------
> WARNING: at net/ipv4/tcp_input.c:3297 tcp_ack+0xd58/0xeba()
> Modules linked in: dm_mirror dm_multipath dm_mod cfi_cmdset_0002 cfi_util
> button jedec_probe cfi_probe gen_probe ck804xrom i2c_nforce2 mtd chipreg
> map_funcs k8temp sg hwmon pcspkr i2c_core serio_raw pata_amd cciss
> ata_generic pata_acpi sata_nv libata sd_mod scsi_mod raid456 async_xor
> async_memcpy async_tx xor raid1 uhci_hcd ohci_hcd ssb ehci_hcd [last
> unloaded: scsi_wait_scan]
> Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
> 
> Call Trace:
>  <IRQ>  [<ffffffff81034481>] warn_on_slowpath+0x51/0x63
>  [<ffffffff8128cdaf>] dev_queue_xmit+0x25b/0x284
>  [<ffffffff812ab9fa>] ip_queue_xmit+0x2cc/0x31f
>  [<ffffffff812b0e3b>] sk_stream_alloc_skb+0x2f/0xd2
>  [<ffffffff8104a1bb>] getnstimeofday+0x2f/0x83
>  [<ffffffff812bb83d>] tcp_transmit_skb+0x775/0x7a4
>  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
>  [<ffffffff812ba74b>] tcp_rcv_established+0x7b9/0x8ce
>  [<ffffffff812c05cd>] tcp_v4_do_rcv+0x2c2/0x48b
>  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
>  [<ffffffff812ae376>] __inet_lookup_established+0xdf/0x17b
>  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
>  [<ffffffff812a7558>] ip_local_deliver+0xd4/0x18e
>  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
>  [<ffffffff8128a2b7>] netif_receive_skb+0x351/0x372
>  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
>  [<ffffffff8128c2ce>] net_rx_action+0xb6/0x1bf
>  [<ffffffff8103919b>] __do_softirq+0x65/0xce
>  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
>  [<ffffffff8100e544>] do_softirq+0x2c/0x68
>  [<ffffffff810390f2>] irq_exit+0x3f/0x83
>  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
>  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
>  <EOI>  [<ffffffff8100adc8>] default_idle+0x0/0x55
>  [<ffffffff8101c138>] lapic_next_event+0x0/0xa
>  [<ffffffff8100adc8>] default_idle+0x0/0x55
>  [<ffffffff8100adf9>] default_idle+0x31/0x55
>  [<ffffffff8100adf4>] default_idle+0x2c/0x55
>  [<ffffffff8100adc8>] default_idle+0x0/0x55
>  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
> 
> ---[ end trace aae73dd976dfcd54 ]---
> TCP wq(s)  S                <
> TCP wq(h) ++h-----+-----+---<
> l0 s1 f2 p18 seq: wq1288576867, su1288576867 hs1288579763 sn1288602883

Aha, it seems you got TCP into the invalid state which I recently added 
check for :-), I'll try to figure out how that could still happen (I think 
I tried to find such code path already earlier but it seems that there's 
still something I've overlooked). Though this is not directly going to 
cause the 2539 WARNING, yet it could, after some other (probably rare 
condition) possibly lead to that as well if this invariant is assumed to 
hold while doing some state manipulation elsewhere in TCP (though I 
think that's not too likely). In case you don't see them too often, you 
can well continue with the patch in order to find the cause for 2539 as 
well (and just ignore those occassional net/ipv4/tcp_input.c:3297 ones for 
now).

Thanks,
Comment 11 Brian Vowell 2008-05-23 11:26:43 UTC 
>
> Aha, it seems you got TCP into the invalid state which I recently added
> check for :-), I'll try to figure out how that could still happen (I think
> I tried to find such code path already earlier but it seems that there's
> still something I've overlooked). Though this is not directly going to
> cause the 2539 WARNING, yet it could, after some other (probably rare
> condition) possibly lead to that as well if this invariant is assumed to
> hold while doing some state manipulation elsewhere in TCP (though I
> think that's not too likely). In case you don't see them too often, you
> can well continue with the patch in order to find the cause for 2539 as
> well (and just ignore those occassional net/ipv4/tcp_input.c:3297 ones for
> now).


Do you want me to continue emailing these traces as they appear in my logs?
<div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Aha, it seems you got TCP into the invalid state which I recently added<br> check for :-), I&#39;ll try to figure out how that could still happen (I think<br>
 I tried to find such code path already earlier but it seems that there&#39;s<br> still something I&#39;ve overlooked). Though this is not directly going to<br> cause the 2539 WARNING, yet it could, after some other (probably rare<br>
 condition) possibly lead to that as well if this invariant is assumed to<br> hold while doing some state manipulation elsewhere in TCP (though I<br> think that&#39;s not too likely). In case you don&#39;t see them too often, you<br>
 can well continue with the patch in order to find the cause for 2539 as<br> well (and just ignore those occassional net/ipv4/tcp_input.c:3297 ones for<br> now).</blockquote><div><br>Do you want me to continue emailing these traces as they appear in my logs? <br>
</div><br></div><br>
Comment 12 Ilpo Järvinen 2008-05-24 01:58:02 UTC 
On Fri, 23 May 2008, Brian Vowell wrote:

> > Aha, it seems you got TCP into the invalid state which I recently added
> > check for :-), I'll try to figure out how that could still happen (I think
> > I tried to find such code path already earlier but it seems that there's
> > still something I've overlooked). Though this is not directly going to
> > cause the 2539 WARNING, yet it could, after some other (probably rare
> > condition) possibly lead to that as well if this invariant is assumed to
> > hold while doing some state manipulation elsewhere in TCP (though I
> > think that's not too likely). In case you don't see them too often, you
> > can well continue with the patch in order to find the cause for 2539 as
> > well (and just ignore those occassional net/ipv4/tcp_input.c:3297 ones for
> > now).
> 
> 
> Do you want me to continue emailing these traces as they appear in my logs?

...Yes, in case you get something different than the WARNING at 3297.
I have at least one suspect already which could lead to that 3297 :-).
Comment 13 Brian Vowell 2008-05-24 07:06:43 UTC 
Here are more traces.  They all are warnings at 3297.

May 23 13:01:10 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:01:10 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:01:10 ishtar kernel:
May 23 13:01:10 ishtar kernel: Call Trace:
May 23 13:01:10 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
warn_on_slowpath+0x51/0x63
May 23 13:01:10 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:01:10 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:01:10 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:01:10 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:01:10 ishtar kernel:  [<ffffffff8102774e>]
__wake_up_common+0x46/0x75
May 23 13:01:10 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:01:10 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:01:10 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:01:10 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:01:10 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:01:10 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:01:10 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:01:10 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:01:10 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:01:10 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:01:10 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:01:10 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:01:10 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:01:10 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:01:10 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 23 13:01:10 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:01:10 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:01:10 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:01:10 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:01:10 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:01:10 ishtar kernel:
May 23 13:01:10 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:01:10 ishtar kernel: TCP wq(s)  s           <
May 23 13:01:10 ishtar kernel: TCP wq(h) ++h+---------<
May 23 13:01:10 ishtar kernel: l0 s1 f2 p13 seq: wq2351018139, su2351018139
hs2351020525 sn2351036453


May 23 13:03:42 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:03:42 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:03:42 ishtar kernel:
May 23 13:03:42 ishtar kernel: Call Trace:
May 23 13:03:42 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:03:42 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:03:42 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:03:42 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:03:42 ishtar kernel:  [<ffffffff8128cdaf>]
dev_queue_xmit+0x25b/0x284
May 23 13:03:42 ishtar kernel:  [<ffffffff8104a1bb>]
getnstimeofday+0x2f/0x83
May 23 13:03:42 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:03:42 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:03:42 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:03:42 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:03:42 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:03:42 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:03:42 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:03:42 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:03:42 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:03:42 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:03:42 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:03:42 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:03:42 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:03:42 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:03:42 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:03:42 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:03:42 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:03:42 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:03:42 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:03:42 ishtar kernel:
May 23 13:03:42 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:03:42 ishtar kernel: TCP wq(s)  s<
May 23 13:03:42 ishtar kernel: TCP wq(h) ++<
May 23 13:03:42 ishtar kernel: l0 s1 f2 p2 seq: wq2360808126, su2360808126
hs2360810043 sn2360810043


May 23 13:12:37 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:12:37 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:12:37 ishtar kernel:
May 23 13:12:37 ishtar kernel: Call Trace:
May 23 13:12:37 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:12:37 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:12:37 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:12:37 ishtar kernel:  [<ffffffff8122d718>]
bond_dev_queue_xmit+0x18a/0x191
May 23 13:12:37 ishtar kernel:  [<ffffffff8104a1bb>]
getnstimeofday+0x2f/0x83
May 23 13:12:37 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:12:37 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:12:37 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:12:37 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:12:37 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:12:37 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:12:37 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:12:37 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:12:37 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:12:37 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:12:37 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:12:37 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:12:37 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:12:37 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:12:37 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:12:37 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:12:37 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:12:37 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:12:37 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:12:37 ishtar kernel:
May 23 13:12:37 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:12:37 ishtar kernel: TCP wq(s)      s     <
May 23 13:12:37 ishtar kernel: TCP wq(h) +----+h+---<
May 23 13:12:37 ishtar kernel: l0 s1 f6 p11 seq: wq2393463171, su2393463171
hs2393471569 sn2393478809


May 23 13:20:46 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:20:46 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:20:46 ishtar kernel:
May 23 13:20:46 ishtar kernel: Call Trace:
May 23 13:20:46 ishtar kernel:  [<ffffffff810389c0>] ns_to_timeval+0xa/0x2f
May 23 13:20:46 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:20:46 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:20:46 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:20:46 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:20:46 ishtar kernel:  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
May 23 13:20:46 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:20:46 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:20:46 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:20:46 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:20:46 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:20:46 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:20:46 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:20:46 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:20:46 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:20:46 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:20:46 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:20:46 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:20:46 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:20:46 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:20:46 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:20:46 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:20:46 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:20:46 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:20:46 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:20:46 ishtar kernel:
May 23 13:20:46 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:20:46 ishtar kernel: TCP wq(s)  s           <
May 23 13:20:46 ishtar kernel: TCP wq(h) ++h+---+-+-+-<
May 23 13:20:46 ishtar kernel: l0 s1 f2 p13 seq: wq2423148138, su2423148138
hs2423150978 sn2423166906


May 23 13:20:54 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:20:54 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:20:54 ishtar kernel:
May 23 13:20:54 ishtar kernel: Call Trace:
May 23 13:20:54 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
warn_on_slowpath+0x51/0x63
May 23 13:20:54 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:20:54 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:20:54 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:20:54 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:20:54 ishtar kernel:  [<ffffffff8129a07a>] __qdisc_run+0xf6/0x1c8
May 23 13:20:54 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:20:54 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:20:54 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:20:54 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:20:54 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:20:54 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:20:54 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:20:54 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:20:54 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:20:54 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:20:54 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:20:54 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:20:54 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:20:54 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:20:54 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 23 13:20:54 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:20:54 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:20:54 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:20:54 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:20:54 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:20:54 ishtar kernel:
May 23 13:20:54 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:20:54 ishtar kernel: TCP wq(s)   s   <
May 23 13:20:54 ishtar kernel: TCP wq(h) +-+h+-<
May 23 13:20:54 ishtar kernel: l0 s1 f3 p6 seq: wq2423575052, su2423575052
hs2423579137 sn2423583481


May 23 13:31:32 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:31:32 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:31:32 ishtar kernel:
May 23 13:31:32 ishtar kernel: Call Trace:
May 23 13:31:32 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
warn_on_slowpath+0x51/0x63
May 23 13:31:32 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:31:32 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:31:32 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:31:32 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:31:32 ishtar kernel:  [<ffffffff8122d718>]
bond_dev_queue_xmit+0x18a/0x191
May 23 13:31:32 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:31:32 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:31:32 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:31:32 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:31:32 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:31:32 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:31:32 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:31:32 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:31:32 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:31:32 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:31:32 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:31:32 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:31:32 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:31:32 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:31:32 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 23 13:31:32 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:31:32 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:31:32 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:31:32 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:31:32 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:31:32 ishtar kernel:
May 23 13:31:32 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:31:32 ishtar kernel: TCP wq(s)   s   <
May 23 13:31:32 ishtar kernel: TCP wq(h) +-+h--<
May 23 13:31:32 ishtar kernel: l0 s1 f3 p6 seq: wq2458775927, su2458775927
hs2458778865 sn2458783209


May 23 13:37:58 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:37:58 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:37:58 ishtar kernel:
May 23 13:37:58 ishtar kernel: Call Trace:
May 23 13:37:58 ishtar kernel:  <IRQ>  [<ffffffff81034481>]
warn_on_slowpath+0x51/0x63
May 23 13:37:58 ishtar kernel:  [<ffffffff812d5943>]
tcp_illinois_cong_avoid+0x167/0x1d1
May 23 13:37:58 ishtar kernel:  [<ffffffff812b760a>] tcp_ack+0xd58/0xeba
May 23 13:37:58 ishtar kernel:  [<ffffffff812ba74b>]
tcp_rcv_established+0x7b9/0x8ce
May 23 13:37:58 ishtar kernel:  [<ffffffff812c05cd>]
tcp_v4_do_rcv+0x2c2/0x48b
May 23 13:37:58 ishtar kernel:  [<ffffffff8102774e>]
__wake_up_common+0x46/0x75
May 23 13:37:58 ishtar kernel:  [<ffffffff812ae376>]
__inet_lookup_established+0xdf/0x17b
May 23 13:37:58 ishtar kernel:  [<ffffffff812c2528>] tcp_v4_rcv+0x6a3/0x700
May 23 13:37:58 ishtar kernel:  [<ffffffff812a7558>]
ip_local_deliver+0xd4/0x18e
May 23 13:37:58 ishtar kernel:  [<ffffffff812a7b0d>] ip_rcv+0x4fb/0x53a
May 23 13:37:58 ishtar kernel:  [<ffffffff8128a2b7>]
netif_receive_skb+0x351/0x372
May 23 13:37:58 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:37:58 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:37:58 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:37:58 ishtar kernel:  [<ffffffff8100ce9c>] call_softirq+0x1c/0x28
May 23 13:37:58 ishtar kernel:  [<ffffffff8100e544>] do_softirq+0x2c/0x68
May 23 13:37:58 ishtar kernel:  [<ffffffff810390f2>] irq_exit+0x3f/0x83
May 23 13:37:58 ishtar kernel:  [<ffffffff8100e813>] do_IRQ+0x13e/0x15f
May 23 13:37:58 ishtar kernel:  [<ffffffff8100c221>] ret_from_intr+0x0/0xa
May 23 13:37:58 ishtar kernel:  <EOI>  [<ffffffff8100adc8>]
default_idle+0x0/0x55
May 23 13:37:58 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 23 13:37:58 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:37:58 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:37:58 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:37:58 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:37:58 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:37:58 ishtar kernel:
May 23 13:37:58 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:37:58 ishtar kernel: TCP wq(s)    S           <
May 23 13:37:58 ishtar kernel: TCP wq(h) +--+h--+-+-----<
May 23 13:37:58 ishtar kernel: l0 s1 f4 p15 seq: wq2475611738, su2475611738
hs2475617530 sn2475633458


May 23 13:37:59 ishtar kernel: WARNING: at net/ipv4/tcp_input.c:3297
tcp_ack+0xd58/0xeba()
May 23 13:37:59 ishtar kernel: Pid: 0, comm: swapper Not tainted 2.6.25.4 #1
May 23 13:37:59 ishtar kernel:
May 23 13:37:59 ishtar kernel: Call Trace:
May 23 13:37:59 ishtar kernel:  [<ffffffff81023577>]
kernel_map_pages+0xe8/0xed
May 23 13:37:59 ishtar kernel:  [<ffffffff8102be0b>]
try_to_wake_up+0x1ae/0x1bf
May 23 13:37:59 ishtar kernel:  [<ffffffff8124150d>] tg3_poll+0x588/0x7df
May 23 13:37:59 ishtar kernel:  [<ffffffff8128c2ce>]
net_rx_action+0xb6/0x1bf
May 23 13:37:59 ishtar kernel:  [<ffffffff8103919b>] __do_softirq+0x65/0xce
May 23 13:37:59 ishtar kernel:  [<ffffffff8101c138>]
lapic_next_event+0x0/0xa
May 23 13:37:59 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:37:59 ishtar kernel:  [<ffffffff8100adf9>] default_idle+0x31/0x55
May 23 13:37:59 ishtar kernel:  [<ffffffff8100adf4>] default_idle+0x2c/0x55
May 23 13:37:59 ishtar kernel:  [<ffffffff8100adc8>] default_idle+0x0/0x55
May 23 13:37:59 ishtar kernel:  [<ffffffff8100ae94>] cpu_idle+0x77/0x9a
May 23 13:37:59 ishtar kernel:
May 23 13:37:59 ishtar kernel: ---[ end trace aae73dd976dfcd54 ]---
May 23 13:37:59 ishtar kernel: TCP wq(s)  s<
May 23 13:37:59 ishtar kernel: TCP wq(h) ++<
May 23 13:37:59 ishtar kernel: l0 s1 f2 p2 seq: wq2475669658, su2475669658
hs2475672418 sn2475672418




On 5/24/08, Ilpo J
Comment 14 Ilpo Järvinen 2008-05-26 04:25:09 UTC 
On Sat, 24 May 2008, Brian Vowell wrote:

> Here are more traces.  They all are warnings at 3297.

Hmm, likely fix to that included below (you might not be able to 
reproduce it anymore though because it's very sensitive to network 
behavior, ie., in case the peer you were communicating with isn't
there any more the necessary events simply won't happen). 

There seems to be yet another candidate in tcp_fastretrans_alert() that 
could cause it but it's extremely unlikely to occur in practice. ...I'll 
fix this one separately later on.

And this isn't exactly there problem we were tracking for but it's nice 
that it showed up as well (so you might want to keep the debug patch 
still applied too)... :-)

-- 
 i.


--
[PATCH] [TCP]: Fix inconsistency source (CA_Open only when !tcp_left_out(tp))

It is possible that this skip path causes TCP to end up into an
invalid state where ca_state was left to CA_Open while some
segments already came into sacked_out. If next valid ACK doesn't
contain new SACK information TCP fails to enter into
tcp_fastretrans_alert(). Thus at least high_seq is set
incorrectly to a too high seqno because some new data segments
could be sent in between (and also, limited transmit is not
being correctly invoked there). Reordering in both directions
can easily cause this situation to occur.

I guess we would want to use tcp_moderate_cwnd(tp) there as well
as it may be possible to use this to trigger oversized burst to
network by sending an old ACK with huge amount of SACK info, but
I'm a bit unsure about its effects (mainly to FlightSize), so to
be on the safe side I just currently fixed it minimally to keep
TCP's state consistent (obviously, such nasty ACKs have been
possible this far). Though it seems that FlightSize is already
underestimated by some amount, so probably on the long term we
might want to trigger recovery there too, if appropriate, to make
FlightSize calculation to resemble reality at the time when the
losses where discovered (but such change scares me too much now
and requires some more thinking anyway how to do that as it
likely involves some code shuffling).

This bug was found by Brian Vowell while running my TCP debug
patch to find cause of another TCP issue (fackets_out
miscount).

Signed-off-by: Ilpo J
Comment 15 Ilpo Järvinen 2008-05-26 04:33:13 UTC 
On Fri, 23 May 2008, Brian Vowell wrote:

> Here are some of the others that show in the logs:
> 
> This one repeats four times, and then again multiple times every minute
> afterwards for about 30 minutes.
> May 18 19:31:00 ishtar kernel: pdflush: page allocation failure. order:3,
> mode:0x4020

I think that the problem here is that tg3 tries to allocate a large 
contiguous memory block which isn't available.

> May 18 19:31:00 ishtar kernel: Pid: 15300, comm: pdflush Not tainted
> 2.6.25.4 #1
> May 18 19:31:00 ishtar kernel:
> May 18 19:31:00 ishtar kernel: Call Trace:
> May 18 19:31:00 ishtar kernel:  <IRQ>  [<ffffffff81075d35>]
> __alloc_pages+0x33f/0x35a
> May 18 19:31:00 ishtar kernel:  [<ffffffff810934a4>] new_slab+0x3f/0x242
> May 18 19:31:00 ishtar kernel:  [<ffffffff81093bba>]
> __slab_alloc+0x212/0x43b
> May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 18 19:31:00 ishtar kernel:  [<ffffffff81094bd6>]
> __kmalloc_node_track_caller+0x75/0xab
> May 18 19:31:00 ishtar kernel:  [<ffffffff8128584a>] __alloc_skb+0x6a/0x133
> May 18 19:31:00 ishtar kernel:  [<ffffffff8128654c>]
> __netdev_alloc_skb+0x29/0x43
> May 18 19:31:00 ishtar kernel:  [<ffffffff8123a405>]
> tg3_alloc_rx_skb+0xc4/0x153
> May 18 19:31:00 ishtar kernel:  [<ffffffff81241343>] tg3_poll+0x3be/0x7df


...but I'm probably not much of help with it.
Comment 16 David S. Miller 2008-06-03 16:35:50 UTC 
From: "Ilpo_J
Comment 17 Ilpo Järvinen 2008-06-04 00:12:41 UTC 
On Tue, 3 Jun 2008, David Miller wrote:

> From: "Ilpo_J
Comment 18 David S. Miller 2008-06-04 11:20:47 UTC 
From: "Ilpo_J
Note You need to log in before you can comment on or make changes to this bug.