The page allocation errors are normal behaviour.  You can reduce their
frequency by increasing /proc/sys/vm/min_free_kbytes

The crash is a bug in drivers/net/mv643xx_eth.c - rxq_refill() is doing add_timer()
on an already-pending timer.  I assume this is a regression added by 8a578111e.

Can you confirm that 2.6.26 was OK?

Reassigned to Lennert, marked as a net driver bug.

(In reply to comment #0)
> Latest working kernel version:
> Earliest failing kernel version:
> Distribution: Debian
> Hardware Environment: TS-7800
> Software Environment: Debian Lenny Armel
> Problem Description: Kernel panic on high CPU load
> 
> Steps to reproduce: netio -u <server>
> 
> ./netio -t 192.168.0.2
> 
> NETIO - Network Throughput Benchmark, Version 1.26
> (C) 1997-2005 Kai Uwe Rommel
> 
> TCP connection established.
> Packet size  1k bytes:  9661 KByte/s Tx,  13466 KByte/s Rx.
> Packet size  2k bytes:  9875 KByte/s Tx,  15276 KByte/s Rx.
> Packet size  4k bytes:  16807 KByte/s Tx,  17449 KByte/s Rx.
> Packet size  8k bytes:  19488 KByte/s Tx,  21989 KByte/s Rx.
> Packet size 16k bytes:  25370 KByte/s Tx,  25335 KByte/s Rx.
> Packet size 32k bytes:  28527 KByte/s Tx,  26968 KByte/s Rx.
> Done.
> 
> ts7800:~/soft/netio# ./netio -u 192.168.0.2
> 
> NETIO - Network Throughput Benchmark, Version 1.26
> (C) 1997-2005 Kai Uwe Rommel
> 
> UDP connection established.
> Packet size  1k bytes:  10566 KByte/s (0%) Tx, ksoftirqd/0: page allocation
> failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c6bc9a80 r7:00000a30 r6:000000a3 r5:ffc4aa30 r4:06bcc020
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bd754>]
> (mv643xx_eth_poll+0x2d4/0x4d4)
> [<c01bd480>] (mv643xx_eth_poll+0x0/0x4d4) from [<c028e05c>]
> (net_rx_action+0x170/0x298)
> [<c028deec>] (net_rx_action+0x0/0x298) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> ksoftirqd/0: page allocation failure. order:0, mode:0x20
> [<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>]
> (__alloc_pages_internal+0x358/0x438)
> [<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>]
> (cache_alloc_refill+0x370/0x6b8)
> [<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>]
> (__kmalloc+0xd0/0xdc)
> [<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
>  r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
> [<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>]
> (dev_alloc_skb+0x20/0x44)
>  r8:c7820000 r7:c01bcdfc r6:c7821f30 r5:00000101 r4:c0408440
> [<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>]
> (rxq_refill+0x128/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bce0c>]
> (rxq_refill_timer_wrapper+0x10/0x14)
> [<c01bcdfc>] (rxq_refill_timer_wrapper+0x0/0x14) from [<c004d4f4>]
> (run_timer_softirq+0x1cc/0x278)
> [<c004d328>] (run_timer_softirq+0x0/0x278) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Mem-info:
> DMA per-cpu:
> CPU    0: hi:   42, btch:   7 usd:   6
> Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
>  free:130 slab:26474 mapped:712 pagetables:56 bounce:0
> DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB
> inactive:6324kB
> present:130048kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0
> DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB
> 0*4096kB = 520kB
> 3417 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap  = 1662716kB
> Total swap = 1662716kB
> 32768 pages of RAM
> 231 free pages
> 1383 reserved pages
> 26474 slab pages
> 3409 pages shared
> 0 pages swap cached
> kernel BUG at include/linux/timer.h:165!
> Unable to handle kernel NULL pointer dereference at virtual address 00000000
> pgd = c0004000
> [00000000] *pgd=00000000
> Internal error: Oops: 817 [#1] PREEMPT
> Modules linked in:
> CPU: 0    Not tainted  (2.6.27-rc3-git6 #14)
> PC is at __bug+0x20/0x2c
> LR is at vprintk+0x314/0x3e8
> pc : [<c002dda4>]    lr : [<c0043324>]    psr: 20000093
> sp : c7821e90  ip : c7821df0  fp : c7821e9c
> r10: c78ce480  r9 : 000005f4  r8 : 00000000
> r7 : 2fa42c56  r6 : c78ce000  r5 : c78ce480  r4 : 20000013
> r3 : 00000000  r2 : c7820000  r1 : c7820000  r0 : 0000002c
> Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
> Control: b005317f  Table: 07380000  DAC: 00000017
> Process ksoftirqd/0 (pid: 3, stack limit = 0xc7820268)
> Stack: (0xc7821e90 to 0xc7822000)
> 1e80:                                     c7821ed4 c7821ea0 c01bc5d8 c002dd94 
> 1ea0: c78ce380 60000013 c7821ed4 20000013 c78ce480 c78ce000 2fa42c56 c7780860 
> 1ec0: ffc4b0a0 00000040 c7821f2c c7821ed8 c01bd754 c01bc430 c78ce380 00000040 
> 1ee0: c78ce45c 00000040 c78ce380 00000000 c7820000 c78ce380 c78ce1dc 00000000 
> 1f00: c7821f1c 00000040 c78ce45c 00000004 00000000 0000012c c041a7a0 c7820000 
> 1f20: c7821f64 c7821f30 c028e05c c01bd490 c7821f30 ffffd4cd c7821f7c c04082a0 
> 1f40: 00000001 00000004 c0408260 00000001 00000000 c040aa68 c7821f94 c7821f68 
> 1f60: c00481a8 c028defc 00000000 60000013 c7820000 c0408260 00000000 00000001 
> 1f80: 00000000 00000000 c7821fac c7821f98 c00482dc c0048144 00000000 00000000 
> 1fa0: c7821fd4 c7821fb0 c0048358 c0048294 00000000 c7820000 00000000 c00482e8 
> 1fc0: c0409774 00000000 c7821ff4 c7821fd8 c0059370 c00482f8 00000000 00000000 
> 1fe0: 00000000 00000000 00000000 c7821ff8 c0045e80 c0059328 25da5a25 a55e49a5 
> Backtrace: 
> [<c002dd84>] (__bug+0x0/0x2c) from [<c01bc5d8>] (rxq_refill+0x1b8/0x1c0)
> [<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bd754>]
> (mv643xx_eth_poll+0x2d4/0x4d4)
> [<c01bd480>] (mv643xx_eth_poll+0x0/0x4d4) from [<c028e05c>]
> (net_rx_action+0x170/0x298)
> [<c028deec>] (net_rx_action+0x0/0x298) from [<c00481a8>]
> (__do_softirq+0x74/0xe8)
> [<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
> [<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
>  r4:00000000
> [<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
>  r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
> [<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
>  r7:00000000 r6:00000000 r5:00000000 r4:00000000
> Code: e1a01000 e59f000c eb0b71e0 e3a03000 (e5833000) 
> Kernel panic - not syncing: Fatal exception in interrupt
> 

(In reply to comment #1)
> The page allocation errors are normal behaviour.  You can reduce their
> frequency by increasing /proc/sys/vm/min_free_kbytes
> 
> The crash is a bug in drivers/net/mv643xx_eth.c - rxq_refill() is doing
> add_timer()
> on an already-pending timer.  I assume this is a regression added by
> 8a578111e.
> 
> Can you confirm that 2.6.26 was OK?
> 
> Reassigned to Lennert, marked as a net driver bug.
> 

Copied driver from 2.6.26. Similar behaviour:
ts7800:~/soft/netio# ./netio -u 192.168.0.2

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

UDP connection established.
Packet size  1k bytes:  10566 KByte/s (0%) Tx, ksoftirqd/0: page allocation failure. order:0, mode:0x20
[<c030a3e0>] (dump_stack+0x0/0x14) from [<c00797b4>] (__alloc_pages_internal+0x358/0x438)
[<c007945c>] (__alloc_pages_internal+0x0/0x438) from [<c0098444>] (cache_alloc_refill+0x370/0x6b8)
[<c00980d4>] (cache_alloc_refill+0x0/0x6b8) from [<c009885c>] (__kmalloc+0xd0/0xdc)
[<c009878c>] (__kmalloc+0x0/0xdc) from [<c0289af8>] (__alloc_skb+0x54/0xf8)
 r7:c7807c20 r6:00000640 r5:00000020 r4:c6bc99c0
[<c0289aa4>] (__alloc_skb+0x0/0xf8) from [<c028a644>] (dev_alloc_skb+0x20/0x44)
 r8:c6bc9a80 r7:00000a30 r6:000000a3 r5:ffc4aa30 r4:06bcc020
[<c028a624>] (dev_alloc_skb+0x0/0x44) from [<c01bc548>] (rxq_refill+0x128/0x1c0)
[<c01bc420>] (rxq_refill+0x0/0x1c0) from [<c01bd754>] (mv643xx_eth_poll+0x2d4/0x4d4)
[<c01bd480>] (mv643xx_eth_poll+0x0/0x4d4) from [<c028e05c>] (net_rx_action+0x170/0x298)
[<c028deec>] (net_rx_action+0x0/0x298) from [<c00481a8>] (__do_softirq+0x74/0xe8)
[<c0048134>] (__do_softirq+0x0/0xe8) from [<c00482dc>] (do_softirq+0x58/0x64)
[<c0048284>] (do_softirq+0x0/0x64) from [<c0048358>] (ksoftirqd+0x70/0x150)
 r4:00000000
[<c00482e8>] (ksoftirqd+0x0/0x150) from [<c0059370>] (kthread+0x58/0x8c)
 r8:00000000 r7:c0409774 r6:c00482e8 r5:00000000 r4:c7820000
[<c0059318>] (kthread+0x0/0x8c) from [<c0045e80>] (do_exit+0x0/0x88c)
 r7:00000000 r6:00000000 r5:00000000 r4:00000000
Mem-info:
DMA per-cpu:
CPU    0: hi:   42, btch:   7 usd:   6
Active:2470 inactive:1581 dirty:6 writeback:0 unstable:0
 free:130 slab:26474 mapped:712 pagetables:56 bounce:0
DMA free:520kB min:1440kB low:1800kB high:2160kB active:9880kB inactive:6324kB present:130048kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 2*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 520kB
3417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 1662716kB
Total swap = 1662716kB
32768 pages of RAM
231 free pages
1383 reserved pages
26474 slab pages
3409 pages shared
0 pages swap cached
...

Created attachment 17411 [details]
Proposed patch.

Sorry for the late response, I've been traveling.

The 'page allocation failure' messages are expected.  Do you also get
the kernel panics in 2.6.26, or just the allocation failure messages?

Can you try whether the attached patch makes the kernel panics go away
for you in 2.6.27?

Created attachment 17436 [details]
Test result using proposed patch against stock 2.6.27-rc4

Created attachment 17437 [details]
Test result using git://git.marvell.com /mv643xx_eth.git Sun, 24 Aug 2008 01:33:44 +0000

Thanks Lennert for your response. I really appreciate your help.

I can't use 2.6.26, because there is no support for the TS7800 board.

With your mentioned patch I still get a kernel panic with 2.6.27-rc4, but with
all your patches form your git repository I only get page allocation failure.
Board is working fine after exception.   

Thanks for the testing.

The log of 2.6.27-rc4 plus the proposed patch shows that you are
subsequently hitting another issue, a NULL pointer deref in rxq_process(),
which is fixed by this patch:

     http://marc.info/?l=linux-netdev&m=121954356302080&w=2

The git repository also contains that particular fix (and a couple more),
so that's why it works fine with that but not with 2.6.27-rc4 plus just
the one patch.

Thanks again for the testing -- I've done some more testing as well and
I think it all looks good, so I will send a pull request for these fixes.

The set of fixes referred to in comment #7, which includes a fix for this
bug, have been merged into 2.6.27-rc5, so I'm closing this bug. (If -rc5
still gives you trouble, please reopen.)