Bug 10324
| Summary: | kernel panic ip_route_input | ||
|---|---|---|---|
| Product: | Networking | Reporter: | Denys Fedoryshchenko (nuclearcat) |
| Component: | IPV4 | Assignee: | Stephen Hemminger (stephen) |
| Status: | CLOSED PATCH_ALREADY_AVAILABLE | ||
| Severity: | normal | CC: | rjw |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 2.6.25-rc6-git6 | Subsystem: | |
| Regression: | Yes | Bisected commit-id: | |
| Bug Depends on: | |||
| Bug Blocks: | 9832 | ||
|
Description
Denys Fedoryshchenko
2008-03-25 12:48:36 UTC
Reply-To: akpm@linux-foundation.org (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Tue, 25 Mar 2008 12:48:37 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=10324 > > Summary: kernel panic ip_route_input > Product: Networking > Version: 2.5 > KernelVersion: 2.6.25-rc6-git6 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: IPV4 > AssignedTo: shemminger@linux-foundation.org > ReportedBy: nuclearcat@nuclearcat.com > > > Latest working kernel version: 2.6.24.x > Earliest failing kernel version: 2.6.25-rc6 A post-2.6.24 regression. > Distribution: Custom > Hardware Environment: Dual CPU, e100 + e1000 > Software Environment: Custom build. FIB_TRIE enabled > Problem Description: Kernel panic after few seconds running network on heavy > traffic (around 350 Mbps). > > Steps to reproduce: It is complicated configuration. I can provide whole > image > of system, around 128MB (it is semi-embedded), but it will need at least few > traffic generators similar to mine. So it is difficult to reproduce. I can > apply any patches required, but highly prefferable if they dont make server > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog > and > software watchdog enabled. > I think it is critical to fix this bug before releasing 2.6.25. > > Here is latest message with panic: > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - > enabled. > > It reboots recently after bringing up networking. Sometimes even not reaching > place where i am loading netconsole. > > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy > to reproduce, but i cannot do bisect, cause this machine dont have power > switch and in difficult to reach area. > I've snipped the log output - it was such a wordwrapped mess :( Can you please resend it in a reply-to-all to this email, after ensuring that it won't be wordwrapped? Thanks. I upload whole netconsole file at http://www.nuclearcat.com/files/ip_route_input_crash.txt There is two crashes. On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote > (switched to email. Please respond via emailed reply-to-all, not > via the bugzilla web interface). > > On Tue, 25 Mar 2008 12:48:37 -0700 (PDT) > bugme-daemon@bugzilla.kernel.org wrote: > > > http://bugzilla.kernel.org/show_bug.cgi?id=10324 > > > > Summary: kernel panic ip_route_input > > Product: Networking > > Version: 2.5 > > KernelVersion: 2.6.25-rc6-git6 > > Platform: All > > OS/Version: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: IPV4 > > AssignedTo: shemminger@linux-foundation.org > > ReportedBy: nuclearcat@nuclearcat.com > > > > > > Latest working kernel version: 2.6.24.x > > Earliest failing kernel version: 2.6.25-rc6 > > A post-2.6.24 regression. > > > Distribution: Custom > > Hardware Environment: Dual CPU, e100 + e1000 > > Software Environment: Custom build. FIB_TRIE enabled > > Problem Description: Kernel panic after few seconds running network on > heavy > > traffic (around 350 Mbps). > > > > Steps to reproduce: It is complicated configuration. I can provide whole > image > > of system, around 128MB (it is semi-embedded), but it will need at least > few > > traffic generators similar to mine. So it is difficult to reproduce. I can > > apply any patches required, but highly prefferable if they dont make server > > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog > and > > software watchdog enabled. > > I think it is critical to fix this bug before releasing 2.6.25. > > > > Here is latest message with panic: > > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW > - > > enabled. > > > > It reboots recently after bringing up networking. Sometimes even not > reaching > > place where i am loading netconsole. > > > > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy > > to reproduce, but i cannot do bisect, cause this machine dont have power > > switch and in difficult to reach area. > > > > I've snipped the log output - it was such a wordwrapped mess :( > > Can you please resend it in a reply-to-all to this email, after ensuring > that it won't be wordwrapped? > > Thanks. -- Denys Fedoryshchenko Technical Manager Virtual ISP S.A.L. Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
recent patches to FIB_TRIE broke something (i have complicated setup with ifb,
maybe it will help). If you can point me to any patch, i can reverse it and
try again with TRIE.
On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote
> (switched to email. Please respond via emailed reply-to-all, not
> via the bugzilla web interface).
>
> On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
>
> > http://bugzilla.kernel.org/show_bug.cgi?id=10324
> >
> > Summary: kernel panic ip_route_input
> > Product: Networking
> > Version: 2.5
> > KernelVersion: 2.6.25-rc6-git6
> > Platform: All
> > OS/Version: Linux
> > Tree: Mainline
> > Status: NEW
> > Severity: normal
> > Priority: P1
> > Component: IPV4
> > AssignedTo: shemminger@linux-foundation.org
> > ReportedBy: nuclearcat@nuclearcat.com
> >
> >
> > Latest working kernel version: 2.6.24.x
> > Earliest failing kernel version: 2.6.25-rc6
>
> A post-2.6.24 regression.
>
> > Distribution: Custom
> > Hardware Environment: Dual CPU, e100 + e1000
> > Software Environment: Custom build. FIB_TRIE enabled
> > Problem Description: Kernel panic after few seconds running network on
> heavy
> > traffic (around 350 Mbps).
> >
> > Steps to reproduce: It is complicated configuration. I can provide whole
> image
> > of system, around 128MB (it is semi-embedded), but it will need at least
> few
> > traffic generators similar to mine. So it is difficult to reproduce. I can
> > apply any patches required, but highly prefferable if they dont make server
> > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog
> and
> > software watchdog enabled.
> > I think it is critical to fix this bug before releasing 2.6.25.
> >
> > Here is latest message with panic:
> > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW
> -
> > enabled.
> >
> > It reboots recently after bringing up networking. Sometimes even not
> reaching
> > place where i am loading netconsole.
> >
> > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy
> > to reproduce, but i cannot do bisect, cause this machine dont have power
> > switch and in difficult to reach area.
> >
>
> I've snipped the log output - it was such a wordwrapped mess :(
>
> Can you please resend it in a reply-to-all to this email, after ensuring
> that it won't be wordwrapped?
>
> Thanks.
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
From: "Denys" <nuclearcat@nuclearcat.com> Date: Wed, 26 Mar 2008 01:12:01 +0200 > Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems > recent patches to FIB_TRIE broke something (i have complicated setup with > ifb, > maybe it will help). If you can point me to any patch, i can reverse it and > try again with TRIE. Unfortunately there were 39 changes made to fib_trie during this time. So it's not like we can give you one patch to try reverting :-/ Really, the fastest way to track this down would be to do a bisect, focused in on net/ipv4/fib_trie.c which git bisect does allow you to do. Reply-To: stephen.hemminger@vyatta.com On Wed, 26 Mar 2008 01:12:01 +0200 "Denys" <nuclearcat@nuclearcat.com> wrote: > Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems > recent patches to FIB_TRIE broke something (i have complicated setup with > ifb, > maybe it will help). If you can point me to any patch, i can reverse it and > try again with TRIE. Bisect? I have to wait when power switch will be installed in that location then. On Tue, 25 Mar 2008 16:49:09 -0700, Stephen Hemminger wrote > On Wed, 26 Mar 2008 01:12:01 +0200 > "Denys" <nuclearcat@nuclearcat.com> wrote: > > > Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems > > recent patches to FIB_TRIE broke something (i have complicated setup with ifb, > > maybe it will help). If you can point me to any patch, i can reverse it and > > try again with TRIE. > > Bisect? > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Denys Fedoryshchenko Technical Manager Virtual ISP S.A.L. It seems that issue is fixed somehow, probably it was related to icmp dst leak, which was fixed in same time. I am not absolutely sure, but it is not crashing as before (after few seconds), system running more than 10 minutes with TRIE, and still alive. I'm closing the bug, then. Please reopen if the issue reappears. |