Bug 30462

Summary: High cpu usage when someone sends many ipv6 udp packages
Product: Networking Reporter: Ernst Persson (ernstp)
Component: IPV6Assignee: Hideaki YOSHIFUJI (yoshfuji)
Status: CLOSED CODE_FIX    
Severity: high CC: florian, maciej.rutecki, rjw
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.38-rc3 Subsystem:
Regression: Yes Bisected commit-id:
Bug Depends on:    
Bug Blocks: 27352    

Description Ernst Persson 2011-03-05 01:10:49 UTC
Hi,

while testing ipv6 performance on my local network I found the following problem.
When sending lots of UDP packages over ipv6 to a machine running kernel 2.6.38-rc3 or later that machines become heavily loaded and X may stop responding for long periods of time for example, sound skips etc.
A ksoftirqd and kworker thread each run 100% on my machine.

I'm using this command to trigger this:
iperf -c fe80::21f:d0ff:fe54:1ad4%eth0 -b 1000M -u -V
You _don't_ have to setup an iperf server on the reciever.

I have a gigabit network with a WNDR3700 with the new V1.0.7.98 firmware that's "IPv6 ready".

I've used the kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/ to narrow down that this happens with 2.6.38-rc3 and later but not with 2.6.38-rc2 and earlier like 2.6.37 and 2.6.35.

The following ipv6 related changes were introduced between -rc2 and -rc3 that I can see. The "Revert 'administrative down' address handling changes." looked big...
      ipv6: Always clone offlink routes.
      ipv6: Revert 'administrative down' address handling changes.
Comment 1 Andrew Morton 2011-03-05 01:16:20 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sat, 5 Mar 2011 01:10:53 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=30462
> 
>            Summary: High cpu usage when someone sends many ipv6 udp
>                     packages
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 2.6.38-rc3
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: high
>           Priority: P1
>          Component: IPV6
>         AssignedTo: yoshfuji@linux-ipv6.org
>         ReportedBy: ernstp@gmail.com
>         Regression: Yes
> 
> 
> Hi,
> 
> while testing ipv6 performance on my local network I found the following
> problem.
> When sending lots of UDP packages over ipv6 to a machine running kernel
> 2.6.38-rc3 or later that machines become heavily loaded and X may stop
> responding for long periods of time for example, sound skips etc.
> A ksoftirqd and kworker thread each run 100% on my machine.
> 
> I'm using this command to trigger this:
> iperf -c fe80::21f:d0ff:fe54:1ad4%eth0 -b 1000M -u -V
> You _don't_ have to setup an iperf server on the reciever.
> 
> I have a gigabit network with a WNDR3700 with the new V1.0.7.98 firmware
> that's
> "IPv6 ready".
> 
> I've used the kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/ to
> narrow down that this happens with 2.6.38-rc3 and later but not with
> 2.6.38-rc2
> and earlier like 2.6.37 and 2.6.35.
> 
> The following ipv6 related changes were introduced between -rc2 and -rc3 that
> I
> can see. The "Revert 'administrative down' address handling changes." looked
> big...
>       ipv6: Always clone offlink routes.
>       ipv6: Revert 'administrative down' address handling changes.
>
Comment 2 David S. Miller 2011-03-05 01:21:43 UTC
From: Andrew Morton <akpm@linux-foundation.org>
Date: Fri, 4 Mar 2011 17:15:06 -0800

>> The following ipv6 related changes were introduced between -rc2 and -rc3
>> that I
>> can see. The "Revert 'administrative down' address handling changes." looked
>> big...
>>       ipv6: Always clone offlink routes.
>>       ipv6: Revert 'administrative down' address handling changes.

Can we narrow it down to which of those two changes introduced the
regression?

We have another issue, still open, which is caused by the first
change, so maybe try reverting that one first.

Thanks.
Comment 3 Ernst Persson 2011-03-05 01:27:04 UTC
I've only used precompiled kernels so far so I don't have stuff set up to
compile kernels right now,
probably won't have time to do that this weekend. Not tonight in GMT+1
anyway!

Regards
//Ernst

On Sat, Mar 5, 2011 at 02:21, David Miller <davem@davemloft.net> wrote:

> From: Andrew Morton <akpm@linux-foundation.org>
> Date: Fri, 4 Mar 2011 17:15:06 -0800
>
> >> The following ipv6 related changes were introduced between -rc2 and -rc3
> that I
> >> can see. The "Revert 'administrative down' address handling changes."
> looked
> >> big...
> >>       ipv6: Always clone offlink routes.
> >>       ipv6: Revert 'administrative down' address handling changes.
>
> Can we narrow it down to which of those two changes introduced the
> regression?
>
> We have another issue, still open, which is caused by the first
> change, so maybe try reverting that one first.
>
> Thanks.
>
Comment 4 Ernst Persson 2011-03-07 19:54:43 UTC
Hi,

seems like reverting "ipv6: Always clone offlink routes." fixes the issue.

Regards
//Ernst

2011/3/5 Ernst Sjöstrand <ernstp@gmail.com>:
> I've only used precompiled kernels so far so I don't have stuff set up to
> compile kernels right now,
> probably won't have time to do that this weekend. Not tonight in GMT+1
> anyway!
>
> Regards
> //Ernst
>
> On Sat, Mar 5, 2011 at 02:21, David Miller <davem@davemloft.net> wrote:
>>
>> From: Andrew Morton <akpm@linux-foundation.org>
>> Date: Fri, 4 Mar 2011 17:15:06 -0800
>>
>> >> The following ipv6 related changes were introduced between -rc2 and
>> >> -rc3 that I
>> >> can see. The "Revert 'administrative down' address handling changes."
>> >> looked
>> >> big...
>> >>       ipv6: Always clone offlink routes.
>> >>       ipv6: Revert 'administrative down' address handling changes.
>>
>> Can we narrow it down to which of those two changes introduced the
>> regression?
>>
>> We have another issue, still open, which is caused by the first
>> change, so maybe try reverting that one first.
>>
>> Thanks.
>
>
Comment 5 Florian Mickler 2011-03-14 17:23:24 UTC
There was a commit merged for 2.6.38-rc9 (or final) which mentions this bug report:

commit 7343ff31ebf01691ea4515d3126467434b9d22d6
Author: David S. Miller <davem@davemloft.net>
Date:   Wed Mar 9 19:55:25 2011 -0800

    ipv6: Don't create clones of host routes.