Bug 10324 - kernel panic ip_route_input
Summary: kernel panic ip_route_input
Status: CLOSED PATCH_ALREADY_AVAILABLE
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks: 9832
  Show dependency tree
 
Reported: 2008-03-25 12:48 UTC by Denys Fedoryshchenko
Modified: 2008-04-04 02:17 UTC (History)
1 user (show)

See Also:
Kernel Version: 2.6.25-rc6-git6
Subsystem:
Regression: Yes
Bisected commit-id:


Attachments

Description Denys Fedoryshchenko 2008-03-25 12:48:36 UTC
Latest working kernel version: 2.6.24.x
Earliest failing kernel version: 2.6.25-rc6
Distribution: Custom
Hardware Environment: Dual CPU, e100 + e1000
Software Environment: Custom build. FIB_TRIE enabled
Problem Description: Kernel panic after few seconds running network on heavy traffic (around 350 Mbps). 

Steps to reproduce: It is complicated configuration. I can provide whole image of system, around 128MB (it is semi-embedded), but it will need at least few traffic generators similar to mine. So it is difficult to reproduce. I can apply any patches required, but highly prefferable if they dont make server unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and software watchdog enabled.
I think it is critical to fix this bug before releasing 2.6.25.

Here is latest message with panic:
CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - 
enabled.

It reboots recently after bringing up networking. Sometimes even not reaching 
place where i am loading netconsole.

Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
to reproduce, but i cannot do bisect, cause this machine dont have power 
switch and in difficult to reach area.

Mar 23 21:53:08 ROUTER [    6.986563] BUG: unable to handle kernel
Mar 23 21:53:08 ROUTER paging request
Mar 23 21:53:08 ROUTER at c0106010
Mar 23 21:53:08 ROUTER [    6.986660] IP:
Mar 23 21:53:08 ROUTER [<c027706a>] ip_route_input+0x872/0xbd9
Mar 23 21:53:08 ROUTER [    6.986726] Oops: 0003 [#1]
Mar 23 21:53:08 ROUTER SMP
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986814] Modules linked in:
Mar 23 21:53:08 ROUTER nf_conntrack_proto_gre
Mar 23 21:53:08 ROUTER nf_nat_proto_gre
Mar 23 21:53:08 ROUTER netconsole
Mar 23 21:53:08 ROUTER configfs
Mar 23 21:53:08 ROUTER hangcheck_timer
Mar 23 21:53:08 ROUTER xt_tcpudp
Mar 23 21:53:08 ROUTER ipt_TTL
Mar 23 21:53:08 ROUTER ipt_ttl
Mar 23 21:53:08 ROUTER xt_NOTRACK
Mar 23 21:53:08 ROUTER iptable_raw
Mar 23 21:53:08 ROUTER iptable_mangle
Mar 23 21:53:08 ROUTER ifb
Mar 23 21:53:08 ROUTER e1000e
Mar 23 21:53:08 ROUTER em_nbyte
Mar 23 21:53:08 ROUTER cls_tcindex
Mar 23 21:53:08 ROUTER act_gact
Mar 23 21:53:08 ROUTER cls_rsvp
Mar 23 21:53:08 ROUTER sch_htb
Mar 23 21:53:08 ROUTER cls_fw
Mar 23 21:53:08 ROUTER act_mirred
Mar 23 21:53:08 ROUTER em_u32
Mar 23 21:53:08 ROUTER sch_red
Mar 23 21:53:08 ROUTER sch_sfq
Mar 23 21:53:08 ROUTER sch_tbf
Mar 23 21:53:08 ROUTER sch_teql
Mar 23 21:53:08 ROUTER cls_basic
Mar 23 21:53:08 ROUTER act_police
Mar 23 21:53:08 ROUTER sch_gred
Mar 23 21:53:08 ROUTER act_pedit
Mar 23 21:53:08 ROUTER sch_hfsc
Mar 23 21:53:08 ROUTER cls_rsvp6
Mar 23 21:53:08 ROUTER sch_ingress
Mar 23 21:53:08 ROUTER em_meta
Mar 23 21:53:08 ROUTER em_text
Mar 23 21:53:08 ROUTER act_ipt
Mar 23 21:53:08 ROUTER sch_dsmark
Mar 23 21:53:08 ROUTER sch_prio
Mar 23 21:53:08 ROUTER sch_netem
Mar 23 21:53:08 ROUTER act_simple
Mar 23 21:53:08 ROUTER cls_u32
Mar 23 21:53:08 ROUTER em_cmp
Mar 23 21:53:08 ROUTER sch_cbq
Mar 23 21:53:08 ROUTER cls_route
Mar 23 21:53:08 ROUTER xt_TCPMSS
Mar 23 21:53:08 ROUTER iptable_nat
Mar 23 21:53:08 ROUTER nf_conntrack_ipv4
Mar 23 21:53:08 ROUTER ipt_LOG
Mar 23 21:53:08 ROUTER ipt_MASQUERADE
Mar 23 21:53:08 ROUTER ipt_REDIRECT
Mar 23 21:53:08 ROUTER nf_nat
Mar 23 21:53:08 ROUTER nf_conntrack
Mar 23 21:53:08 ROUTER nfnetlink
Mar 23 21:53:08 ROUTER iptable_filter
Mar 23 21:53:08 ROUTER ip_tables
Mar 23 21:53:08 ROUTER x_tables
Mar 23 21:53:08 ROUTER 8021q
Mar 23 21:53:08 ROUTER tun
Mar 23 21:53:08 ROUTER tulip
Mar 23 21:53:08 ROUTER r8169
Mar 23 21:53:08 ROUTER sky2
Mar 23 21:53:08 ROUTER via_velocity
Mar 23 21:53:08 ROUTER via_rhine
Mar 23 21:53:08 ROUTER sis900
Mar 23 21:53:08 ROUTER ne2k_pci
Mar 23 21:53:08 ROUTER 8390
Mar 23 21:53:08 ROUTER skge
Mar 23 21:53:08 ROUTER tg3
Mar 23 21:53:08 ROUTER 8139too
Mar 23 21:53:08 ROUTER e1000
Mar 23 21:53:08 ROUTER e100
Mar 23 21:53:08 ROUTER usb_storage
Mar 23 21:53:08 ROUTER mtdblock
Mar 23 21:53:08 ROUTER mtd_blkdevs
Mar 23 21:53:08 ROUTER usbhid
Mar 23 21:53:08 ROUTER uhci_hcd
Mar 23 21:53:08 ROUTER ehci_hcd
Mar 23 21:53:08 ROUTER ohci_hcd
Mar 23 21:53:08 ROUTER usbcore
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957]
Mar 23 21:53:08 ROUTER [    6.986957] Pid: 0, comm: swapper Not tainted 
(2.6.25-rc6-git6-build-0024 #2)
Mar 23 21:53:08 ROUTER [    6.986957] EIP: 0060:[<c027706a>] EFLAGS: 00010282 
CPU: 0
Mar 23 21:53:08 ROUTER [    6.986957] EIP is at ip_route_input+0x872/0xbd9
Mar 23 21:53:08 ROUTER [    6.986957] EAX: c0928100 EBX: 000000c2 ECX: 
c092ee78 EDX: c0105ff8
Mar 23 21:53:08 ROUTER [    6.986957] ESI: ffffff8f EDI: c092ef08 EBP: 
c092ef34 ESP: c092ee98
Mar 23 21:53:08 ROUTER [    6.986957]  DS: 007b ES: 007b FS: 00d8 GS: 0000 
SS: 0068
Mar 23 21:53:08 ROUTER [    6.986957] Process swapper (pid: 0, ti=c092e000 
task=c03763a0 task.ti=c039c000)
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957] Stack:
Mar 23 21:53:08 ROUTER c092eecc
Mar 23 21:53:08 ROUTER 459b92c2
Mar 23 21:53:08 ROUTER ff00a8c0
Mar 23 21:53:08 ROUTER f755b300
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER c092eed8
Mar 23 21:53:08 ROUTER f898e3bf
Mar 23 21:53:08 ROUTER f755b300
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957]
Mar 23 21:53:08 ROUTER 00000001
Mar 23 21:53:08 ROUTER 00000002
Mar 23 21:53:08 ROUTER f7c58800
Mar 23 21:53:08 ROUTER c096e800
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000003
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957]
Mar 23 21:53:08 ROUTER ff00a8c0
Mar 23 21:53:08 ROUTER 459b92c2
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER 00000000
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957] Call Trace:
Mar 23 21:53:08 ROUTER [    6.986957]  [<f898e3bf>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER nf_nat_fn+0x15b/0x16b [iptable_nat]
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0105ff8>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER do_IRQ+0xba/0xd2
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0278898>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER ip_rcv_finish+0x2c/0x29a
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0278ef8>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER ip_rcv+0x202/0x22c
Mar 23 21:53:08 ROUTER [    6.986957]  [<c025ee4e>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER netif_receive_skb+0x33e/0x3a9
Mar 23 21:53:08 ROUTER [    6.986957]  [<c02612c2>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER process_backlog+0x62/0xb5
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0260d27>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER net_rx_action+0x8f/0x191
Mar 23 21:53:08 ROUTER [    6.986957]  [<c01240a7>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER __do_softirq+0x64/0xcd
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0105f0a>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER do_softirq+0x55/0x89
Mar 23 21:53:08 ROUTER [    6.986957]  [<c013f49e>]
Mar 23 21:53:08 ROUTER handle_fasteoi_irq+0x0/0xaa
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0124010>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER irq_exit+0x38/0x6b
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0105ff8>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER do_IRQ+0xba/0xd2
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0104287>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER common_interrupt+0x23/0x28
Mar 23 21:53:08 ROUTER [    6.986957]  [<c01021ba>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER mwait_idle_with_hints+0x3a/0x3e
Mar 23 21:53:08 ROUTER [    6.986957]  [<c01021be>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER mwait_idle+0x0/0xf
Mar 23 21:53:08 ROUTER [    6.986957]  [<c01021cb>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER mwait_idle+0xd/0xf
Mar 23 21:53:08 ROUTER [    6.986957]  [<c0102668>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER cpu_idle+0x8a/0xa6
Mar 23 21:53:08 ROUTER [    6.986957]  [<c02a8b0d>]
Mar 23 21:53:08 ROUTER ?
Mar 23 21:53:08 ROUTER rest_init+0x49/0x4b
Mar 23 21:53:08 ROUTER [    6.986957]  =======================
Mar 23 21:53:08 ROUTER [    6.986957] Code:
Mar 23 21:53:08 ROUTER 84
Mar 23 21:53:08 ROUTER 01
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 8b
Mar 23 21:53:08 ROUTER 55
Mar 23 21:53:08 ROUTER 8c
Mar 23 21:53:08 ROUTER f0
Mar 23 21:53:08 ROUTER ff
Mar 23 21:53:08 ROUTER 4a
Mar 23 21:53:08 ROUTER 04
Mar 23 21:53:08 ROUTER 0f
Mar 23 21:53:08 ROUTER 94
Mar 23 21:53:08 ROUTER c0
Mar 23 21:53:08 ROUTER 84
Mar 23 21:53:08 ROUTER c0
Mar 23 21:53:08 ROUTER 74
Mar 23 21:53:08 ROUTER 07
Mar 23 21:53:08 ROUTER 89
Mar 23 21:53:08 ROUTER d0
Mar 23 21:53:08 ROUTER e8
Mar 23 21:53:08 ROUTER e9
Mar 23 21:53:08 ROUTER f8
Mar 23 21:53:08 ROUTER 01
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 83
Mar 23 21:53:08 ROUTER 7d
Mar 23 21:53:08 ROUTER 84
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 0f
Mar 23 21:53:08 ROUTER 84
Mar 23 21:53:08 ROUTER 64
Mar 23 21:53:08 ROUTER 03
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 8b
Mar 23 21:53:08 ROUTER 55
Mar 23 21:53:08 ROUTER d8
Mar 23 21:53:08 ROUTER 85
Mar 23 21:53:08 ROUTER d2
Mar 23 21:53:08 ROUTER 74
Mar 23 21:53:08 ROUTER 12
Mar 23 21:53:08 ROUTER unparseable log message: "<f0> "
Mar 23 21:53:08 ROUTER ff
Mar 23 21:53:08 ROUTER 4a
Mar 23 21:53:08 ROUTER 18
Mar 23 21:53:08 ROUTER 0f
Mar 23 21:53:08 ROUTER 94
Mar 23 21:53:08 ROUTER c0
Mar 23 21:53:08 ROUTER 84
Mar 23 21:53:08 ROUTER c0
Mar 23 21:53:08 ROUTER 74
Mar 23 21:53:08 ROUTER 07
Mar 23 21:53:08 ROUTER 89
Mar 23 21:53:08 ROUTER d0
Mar 23 21:53:08 ROUTER e8
Mar 23 21:53:08 ROUTER c9
Mar 23 21:53:08 ROUTER 6c
Mar 23 21:53:08 ROUTER 02
Mar 23 21:53:08 ROUTER 00
Mar 23 21:53:08 ROUTER 8b
Mar 23 21:53:08 ROUTER 55
Mar 23 21:53:08 ROUTER dc
Mar 23 21:53:08 ROUTER
Mar 23 21:53:08 ROUTER [    6.986957] EIP: [<c027706a>]
Mar 23 21:53:08 ROUTER ip_route_input+0x872/0xbd9
Mar 23 21:53:08 ROUTER SS:ESP 0068:c092ee98
Mar 23 21:53:08 ROUTER [    6.986957] Kernel panic - not syncing: Fatal 
exception in interrupt
Mar 23 21:53:08 ROUTER [    6.986957] Rebooting in 10 seconds..

One more 

Mar 23 21:56:26 ROUTER [   14.492999] BUG: unable to handle kernel
Mar 23 21:56:26 ROUTER paging request
Mar 23 21:56:26 ROUTER at 00112e2e
Mar 23 21:56:26 ROUTER [   14.493118] IP:
Mar 23 21:56:26 ROUTER [<c029d7d4>] fib_validate_source+0x20a/0x26e
Mar 23 21:56:26 ROUTER [   14.493203] Oops: 0002 [#1]
Mar 23 21:56:26 ROUTER SMP
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493308] Modules linked in:
Mar 23 21:56:26 ROUTER softdog
Mar 23 21:56:26 ROUTER nf_nat_pptp
Mar 23 21:56:26 ROUTER nf_conntrack_pptp
Mar 23 21:56:26 ROUTER nf_conntrack_proto_gre
Mar 23 21:56:26 ROUTER nf_nat_proto_gre
Mar 23 21:56:26 ROUTER netconsole
Mar 23 21:56:26 ROUTER configfs
Mar 23 21:56:26 ROUTER hangcheck_timer
Mar 23 21:56:26 ROUTER xt_tcpudp
Mar 23 21:56:26 ROUTER ipt_TTL
Mar 23 21:56:26 ROUTER ipt_ttl
Mar 23 21:56:26 ROUTER xt_NOTRACK
Mar 23 21:56:26 ROUTER iptable_raw
Mar 23 21:56:26 ROUTER iptable_mangle
Mar 23 21:56:26 ROUTER ifb
Mar 23 21:56:26 ROUTER e1000e
Mar 23 21:56:26 ROUTER em_nbyte
Mar 23 21:56:26 ROUTER cls_tcindex
Mar 23 21:56:26 ROUTER act_gact
Mar 23 21:56:26 ROUTER cls_rsvp
Mar 23 21:56:26 ROUTER sch_htb
Mar 23 21:56:26 ROUTER cls_fw
Mar 23 21:56:26 ROUTER act_mirred
Mar 23 21:56:26 ROUTER em_u32
Mar 23 21:56:26 ROUTER sch_red
Mar 23 21:56:26 ROUTER sch_sfq
Mar 23 21:56:26 ROUTER sch_tbf
Mar 23 21:56:26 ROUTER sch_teql
Mar 23 21:56:26 ROUTER cls_basic
Mar 23 21:56:26 ROUTER act_police
Mar 23 21:56:26 ROUTER sch_gred
Mar 23 21:56:26 ROUTER act_pedit
Mar 23 21:56:26 ROUTER sch_hfsc
Mar 23 21:56:26 ROUTER cls_rsvp6
Mar 23 21:56:26 ROUTER sch_ingress
Mar 23 21:56:26 ROUTER em_meta
Mar 23 21:56:26 ROUTER em_text
Mar 23 21:56:26 ROUTER act_ipt
Mar 23 21:56:26 ROUTER sch_dsmark
Mar 23 21:56:26 ROUTER sch_prio
Mar 23 21:56:26 ROUTER sch_netem
Mar 23 21:56:26 ROUTER act_simple
Mar 23 21:56:26 ROUTER cls_u32
Mar 23 21:56:26 ROUTER em_cmp
Mar 23 21:56:26 ROUTER sch_cbq
Mar 23 21:56:26 ROUTER cls_route
Mar 23 21:56:26 ROUTER xt_TCPMSS
Mar 23 21:56:26 ROUTER iptable_nat
Mar 23 21:56:26 ROUTER nf_conntrack_ipv4
Mar 23 21:56:26 ROUTER ipt_LOG
Mar 23 21:56:26 ROUTER ipt_MASQUERADE
Mar 23 21:56:26 ROUTER ipt_REDIRECT
Mar 23 21:56:26 ROUTER nf_nat
Mar 23 21:56:26 ROUTER nf_conntrack
Mar 23 21:56:26 ROUTER nfnetlink
Mar 23 21:56:26 ROUTER iptable_filter
Mar 23 21:56:26 ROUTER ip_tables
Mar 23 21:56:26 ROUTER x_tables
Mar 23 21:56:26 ROUTER 8021q
Mar 23 21:56:26 ROUTER tun
Mar 23 21:56:26 ROUTER tulip
Mar 23 21:56:26 ROUTER r8169
Mar 23 21:56:26 ROUTER sky2
Mar 23 21:56:26 ROUTER via_velocity
Mar 23 21:56:26 ROUTER via_rhine
Mar 23 21:56:26 ROUTER sis900
Mar 23 21:56:26 ROUTER ne2k_pci
Mar 23 21:56:26 ROUTER 8390
Mar 23 21:56:26 ROUTER skge
Mar 23 21:56:26 ROUTER tg3
Mar 23 21:56:26 ROUTER 8139too
Mar 23 21:56:26 ROUTER e1000
Mar 23 21:56:26 ROUTER e100
Mar 23 21:56:26 ROUTER usb_storage
Mar 23 21:56:26 ROUTER mtdblock
Mar 23 21:56:26 ROUTER mtd_blkdevs
Mar 23 21:56:26 ROUTER usbhid
Mar 23 21:56:26 ROUTER uhci_hcd
Mar 23 21:56:26 ROUTER ehci_hcd
Mar 23 21:56:26 ROUTER ohci_hcd
Mar 23 21:56:26 ROUTER usbcore
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821]
Mar 23 21:56:26 ROUTER [   14.493821] Pid: 0, comm: swapper Not tainted 
(2.6.25-rc6-git6-build-0024 #2)
Mar 23 21:56:26 ROUTER [   14.493821] EIP: 0060:[<c029d7d4>] EFLAGS: 00010202 
CPU: 1
Mar 23 21:56:26 ROUTER [   14.493821] EIP is at fib_validate_source+0x20a/
0x26e
Mar 23 21:56:26 ROUTER [   14.493821] EAX: 00000000 EBX: c092fe4c ECX: 
c092fde0 EDX: 00112e16
Mar 23 21:56:26 ROUTER [   14.493821] ESI: 00000000 EDI: c096e800 EBP: 
c092fe68 ESP: c092fe00
Mar 23 21:56:26 ROUTER [   14.493821]  DS: 007b ES: 007b FS: 00d8 GS: 0000 
SS: 0068
Mar 23 21:56:26 ROUTER [   14.493821] Process swapper (pid: 0, ti=c092f000 
task=f7c28b00 task.ti=f7c1f000)
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821] Stack:
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 6f00000a
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER f71a2680
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 6f00000a
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821]
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821]
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00000000
Mar 23 21:56:26 ROUTER 00112e16
Mar 23 21:56:26 ROUTER f7cad880
Mar 23 21:56:26 ROUTER f7ffbd00
Mar 23 21:56:26 ROUTER 0000000a
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821] Call Trace:
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0277103>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER ip_route_input+0x90b/0xbd9
Mar 23 21:56:26 ROUTER [   14.493821]  [<f898e3bf>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER nf_nat_fn+0x15b/0x16b [iptable_nat]
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0278898>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER ip_rcv_finish+0x2c/0x29a
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0278ef8>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER ip_rcv+0x202/0x22c
Mar 23 21:56:26 ROUTER [   14.493821]  [<c025ee4e>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER netif_receive_skb+0x33e/0x3a9
Mar 23 21:56:26 ROUTER [   14.493821]  [<f88d3a3e>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER e100_poll+0x17e/0x2de [e100]
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01316a3>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER run_hrtimer_pending+0xd/0x84
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0260d27>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER net_rx_action+0x8f/0x191
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01240a7>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER __do_softirq+0x64/0xcd
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0105f0a>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER do_softirq+0x55/0x89
Mar 23 21:56:26 ROUTER [   14.493821]  [<c013f49e>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER handle_fasteoi_irq+0x0/0xaa
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0124010>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER irq_exit+0x38/0x6b
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0105ff8>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER do_IRQ+0xba/0xd2
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0104287>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER common_interrupt+0x23/0x28
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01300d8>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER posix_cpu_timer_set+0x11d/0x35c
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01021ba>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER mwait_idle_with_hints+0x3a/0x3e
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01021be>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER mwait_idle+0x0/0xf
Mar 23 21:56:26 ROUTER [   14.493821]  [<c01021cb>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER mwait_idle+0xd/0xf
Mar 23 21:56:26 ROUTER [   14.493821]  [<c0102668>]
Mar 23 21:56:26 ROUTER ?
Mar 23 21:56:26 ROUTER cpu_idle+0x8a/0xa6
Mar 23 21:56:26 ROUTER [   14.493821]  =======================
Mar 23 21:56:26 ROUTER [   14.493821] Code:
Mar 23 21:56:26 ROUTER e9
Mar 23 21:56:26 ROUTER 80
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 85
Mar 23 21:56:26 ROUTER f6
Mar 23 21:56:26 ROUTER 75
Mar 23 21:56:26 ROUTER 53
Mar 23 21:56:26 ROUTER 8b
Mar 23 21:56:26 ROUTER 45
Mar 23 21:56:26 ROUTER 0c
Mar 23 21:56:26 ROUTER 31
Mar 23 21:56:26 ROUTER c9
Mar 23 21:56:26 ROUTER 31
Mar 23 21:56:26 ROUTER d2
Mar 23 21:56:26 ROUTER e8
Mar 23 21:56:26 ROUTER d4
Mar 23 21:56:26 ROUTER 9f
Mar 23 21:56:26 ROUTER ff
Mar 23 21:56:26 ROUTER ff
Mar 23 21:56:26 ROUTER 8b
Mar 23 21:56:26 ROUTER 4d
Mar 23 21:56:26 ROUTER 10
Mar 23 21:56:26 ROUTER 89
Mar 23 21:56:26 ROUTER 01
Mar 23 21:56:26 ROUTER 8b
Mar 23 21:56:26 ROUTER 5d
Mar 23 21:56:26 ROUTER 14
Mar 23 21:56:26 ROUTER c7
Mar 23 21:56:26 ROUTER 03
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 31
Mar 23 21:56:26 ROUTER db
Mar 23 21:56:26 ROUTER eb
Mar 23 21:56:26 ROUTER 5e
Mar 23 21:56:26 ROUTER 85
Mar 23 21:56:26 ROUTER d2
Mar 23 21:56:26 ROUTER 74
Mar 23 21:56:26 ROUTER 12
Mar 23 21:56:26 ROUTER unparseable log message: "<f0> "
Mar 23 21:56:26 ROUTER ff
Mar 23 21:56:26 ROUTER 4a
Mar 23 21:56:26 ROUTER 18
Mar 23 21:56:26 ROUTER 0f
Mar 23 21:56:26 ROUTER 94
Mar 23 21:56:26 ROUTER c0
Mar 23 21:56:26 ROUTER 84
Mar 23 21:56:26 ROUTER c0
Mar 23 21:56:26 ROUTER 74
Mar 23 21:56:26 ROUTER 07
Mar 23 21:56:26 ROUTER 89
Mar 23 21:56:26 ROUTER d0
Mar 23 21:56:26 ROUTER e8
Mar 23 21:56:26 ROUTER 5f
Mar 23 21:56:26 ROUTER 05
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 00
Mar 23 21:56:26 ROUTER 8b
Mar 23 21:56:26 ROUTER 55
Mar 23 21:56:26 ROUTER ec
Mar 23 21:56:26 ROUTER
Mar 23 21:56:26 ROUTER [   14.493821] EIP: [<c029d7d4>]
Mar 23 21:56:26 ROUTER fib_validate_source+0x20a/0x26e
Mar 23 21:56:26 ROUTER SS:ESP 0068:c092fe00
Mar 23 21:56:26 ROUTER [   14.493822] Kernel panic - not syncing: Fatal 
exception in interrupt
Comment 1 Anonymous Emailer 2008-03-25 13:03:33 UTC
Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=10324
> 
>            Summary: kernel panic ip_route_input
>            Product: Networking
>            Version: 2.5
>      KernelVersion: 2.6.25-rc6-git6
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: IPV4
>         AssignedTo: shemminger@linux-foundation.org
>         ReportedBy: nuclearcat@nuclearcat.com
> 
> 
> Latest working kernel version: 2.6.24.x
> Earliest failing kernel version: 2.6.25-rc6

A post-2.6.24 regression.

> Distribution: Custom
> Hardware Environment: Dual CPU, e100 + e1000
> Software Environment: Custom build. FIB_TRIE enabled
> Problem Description: Kernel panic after few seconds running network on heavy
> traffic (around 350 Mbps). 
> 
> Steps to reproduce: It is complicated configuration. I can provide whole
> image
> of system, around 128MB (it is semi-embedded), but it will need at least few
> traffic generators similar to mine. So it is difficult to reproduce. I can
> apply any patches required, but highly prefferable if they dont make server
> unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog
> and
> software watchdog enabled.
> I think it is critical to fix this bug before releasing 2.6.25.
> 
> Here is latest message with panic:
> CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - 
> enabled.
> 
> It reboots recently after bringing up networking. Sometimes even not reaching 
> place where i am loading netconsole.
> 
> Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> to reproduce, but i cannot do bisect, cause this machine dont have power 
> switch and in difficult to reach area.
> 

I've snipped the log output - it was such a wordwrapped mess :(

Can you please resend it in a reply-to-all to this email, after ensuring
that it won't be wordwrapped?

Thanks.
Comment 2 Denys Fedoryshchenko 2008-03-25 16:07:47 UTC
I upload whole netconsole file at
http://www.nuclearcat.com/files/ip_route_input_crash.txt 
There is two crashes.



On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote
> (switched to email.  Please respond via emailed reply-to-all, not 
> via the bugzilla web interface).
> 
> On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=10324
> > 
> >            Summary: kernel panic ip_route_input
> >            Product: Networking
> >            Version: 2.5
> >      KernelVersion: 2.6.25-rc6-git6
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: IPV4
> >         AssignedTo: shemminger@linux-foundation.org
> >         ReportedBy: nuclearcat@nuclearcat.com
> > 
> > 
> > Latest working kernel version: 2.6.24.x
> > Earliest failing kernel version: 2.6.25-rc6
> 
> A post-2.6.24 regression.
> 
> > Distribution: Custom
> > Hardware Environment: Dual CPU, e100 + e1000
> > Software Environment: Custom build. FIB_TRIE enabled
> > Problem Description: Kernel panic after few seconds running network on
> heavy
> > traffic (around 350 Mbps). 
> > 
> > Steps to reproduce: It is complicated configuration. I can provide whole
> image
> > of system, around 128MB (it is semi-embedded), but it will need at least
> few
> > traffic generators similar to mine. So it is difficult to reproduce. I can
> > apply any patches required, but highly prefferable if they dont make server
> > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog
> and
> > software watchdog enabled.
> > I think it is critical to fix this bug before releasing 2.6.25.
> > 
> > Here is latest message with panic:
> > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW
> - 
> > enabled.
> > 
> > It reboots recently after bringing up networking. Sometimes even not
> reaching 
> > place where i am loading netconsole.
> > 
> > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> > to reproduce, but i cannot do bisect, cause this machine dont have power 
> > switch and in difficult to reach area.
> >
> 
> I've snipped the log output - it was such a wordwrapped mess :(
> 
> Can you please resend it in a reply-to-all to this email, after ensuring
> that it won't be wordwrapped?
> 
> Thanks.


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Comment 3 Denys Fedoryshchenko 2008-03-25 16:12:22 UTC
Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
recent patches to FIB_TRIE broke something (i have complicated setup with ifb,
maybe it will help). If you can point me to any patch, i can reverse it and
try again with TRIE.

On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote
> (switched to email.  Please respond via emailed reply-to-all, not 
> via the bugzilla web interface).
> 
> On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=10324
> > 
> >            Summary: kernel panic ip_route_input
> >            Product: Networking
> >            Version: 2.5
> >      KernelVersion: 2.6.25-rc6-git6
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: IPV4
> >         AssignedTo: shemminger@linux-foundation.org
> >         ReportedBy: nuclearcat@nuclearcat.com
> > 
> > 
> > Latest working kernel version: 2.6.24.x
> > Earliest failing kernel version: 2.6.25-rc6
> 
> A post-2.6.24 regression.
> 
> > Distribution: Custom
> > Hardware Environment: Dual CPU, e100 + e1000
> > Software Environment: Custom build. FIB_TRIE enabled
> > Problem Description: Kernel panic after few seconds running network on
> heavy
> > traffic (around 350 Mbps). 
> > 
> > Steps to reproduce: It is complicated configuration. I can provide whole
> image
> > of system, around 128MB (it is semi-embedded), but it will need at least
> few
> > traffic generators similar to mine. So it is difficult to reproduce. I can
> > apply any patches required, but highly prefferable if they dont make server
> > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog
> and
> > software watchdog enabled.
> > I think it is critical to fix this bug before releasing 2.6.25.
> > 
> > Here is latest message with panic:
> > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW
> - 
> > enabled.
> > 
> > It reboots recently after bringing up networking. Sometimes even not
> reaching 
> > place where i am loading netconsole.
> > 
> > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> > to reproduce, but i cannot do bisect, cause this machine dont have power 
> > switch and in difficult to reach area.
> >
> 
> I've snipped the log output - it was such a wordwrapped mess :(
> 
> Can you please resend it in a reply-to-all to this email, after ensuring
> that it won't be wordwrapped?
> 
> Thanks.


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Comment 4 David S. Miller 2008-03-25 16:45:57 UTC
From: "Denys" <nuclearcat@nuclearcat.com>
Date: Wed, 26 Mar 2008 01:12:01 +0200

> Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
> recent patches to FIB_TRIE broke something (i have complicated setup with
> ifb,
> maybe it will help). If you can point me to any patch, i can reverse it and
> try again with TRIE.

Unfortunately there were 39 changes made to fib_trie
during this time.  So it's not like we can give you
one patch to try reverting :-/

Really, the fastest way to track this down would be
to do a bisect, focused in on net/ipv4/fib_trie.c
which git bisect does allow you to do.
Comment 5 Anonymous Emailer 2008-03-25 16:49:32 UTC
Reply-To: stephen.hemminger@vyatta.com

On Wed, 26 Mar 2008 01:12:01 +0200
"Denys" <nuclearcat@nuclearcat.com> wrote:

> Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
> recent patches to FIB_TRIE broke something (i have complicated setup with
> ifb,
> maybe it will help). If you can point me to any patch, i can reverse it and
> try again with TRIE.

Bisect?
Comment 6 Denys Fedoryshchenko 2008-03-26 01:20:57 UTC
I have to wait when power switch will be installed in that location then.


On Tue, 25 Mar 2008 16:49:09 -0700, Stephen Hemminger wrote
> On Wed, 26 Mar 2008 01:12:01 +0200
> "Denys" <nuclearcat@nuclearcat.com> wrote:
> 
> > Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It 
seems
> > recent patches to FIB_TRIE broke something (i have complicated setup with 
ifb,
> > maybe it will help). If you can point me to any patch, i can reverse it 
and
> > try again with TRIE.
> 
> Bisect?
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Comment 7 Denys Fedoryshchenko 2008-04-03 20:59:18 UTC
It seems that issue is fixed somehow, probably it was related to icmp dst leak, which was fixed in same time.
I am not absolutely sure, but it is not crashing as before (after few seconds), system running more than 10 minutes with TRIE, and still alive.
Comment 8 Rafael J. Wysocki 2008-04-04 02:17:33 UTC
I'm closing the bug, then.  Please reopen if the issue reappears.

Note You need to log in before you can comment on or make changes to this bug.