Bug 9318

Summary: 2.6.23.1-smp kernel panic (network-related)
Product: Networking Reporter: Krzysztof Mościcki (stivi)
Component: OtherAssignee: Arnaldo Carvalho de Melo (acme)
Status: REJECTED INVALID    
Severity: normal CC: jarkao2, marek
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.23.1 Subsystem:
Regression: --- Bisected commit-id:
Attachments: kernel 2.6.23.1 - config
Print timer running function in panic
Print in panic plus verify timer function address

Description Krzysztof Mościcki 2007-11-07 12:03:58 UTC
Most recent kernel where this bug did not occur:
Distribution: gentoo
Hardware Environment: see below
Software Environment: see below
Problem Description:
My company's (ISP) bussines model requires dynamic resizing of the
client queues. It's achieved by regenerating shaping rules and loading
then using batch mode of a tc binary. On production systems it's done
once every 1 or 2 minutes. Unfortunately this causes smp kernels to
panic. Non-smp kernels don't have such problems. Bug is around a long
time. I first noticed it after migrating to shaping configs that use
IFB, it might have been 2.6.18 "era".



Steps to reproduce:
I'm feeding the machine with production traffic
by means of port mirroring. Test machine has the same config as
production one (including mac addresses), so it tries to route the
incoming traffic.

Tested kernels were 2.6.23.1 and 2.6.20.6. Both panicked if compiled with SMP support and work stable otherwise. Problem occurs only with cyclic "shaping restarts". For the test, reload operation using "tc -b ..." was executed in an infinite loop. 

Box's CPU usage was approximately 15%. Panics occur with few hours -
one day intervals.


Below I attach the panic message captured via serial console:
----------------------------------------------------------------------
printk: 63 messages suppressed.
dst cache overflow
SMP
Modules linked in: ipt_LOG xt_hashlimit ipt_MASQUERADE ip_set_macipmap
ip_set_ipmap xt_state w83627hf hwmon_vid eeprom ifb ipt_SET ipt_set
ip_set ipip tunnel4 ip_gre e1000 i2c_i801 i2c_core CPU:    1 EIP:
0060:[<f255c08f>]    Not tainted VLI EFLAGS: 00010202   (2.6.23.1-smp
#2) EIP is at 0xf255c08f
eax: c196a000   ebx: 00000100   ecx: ef2f408c   edx: f3630000
esi: f0332029   edi: f255c08c   ebp: 00000001   esp: f3631ea8
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process tc (pid: 27695, ti=f3630000 task=f26a9560 task.ti=f3630000)
Stack: c01280ad f26a9560 00000001 f3631eb4 c0106f57 ef2f408c f0e8c08c
00000031 c0495308 0000000a c0124eb6 00000046 00000000 f7657740 f3630000
c0124f4c c180f120 c0114e62 00000000 00000000 f7bfd224 f74ed95c c01047e0
f7bfd224 Call Trace:
 [<c01280ad>] run_timer_softirq+0xf5/0x154
 [<c0106f57>] profile_pc+0x21/0x4a
 [<c0124eb6>] __do_softirq+0x5d/0xc1
 [<c0124f4c>] do_softirq+0x32/0x36
 [<c0114e62>] smp_apic_timer_interrupt+0x74/0x80
 [<c01047e0>] apic_timer_interrupt+0x28/0x30
 [<c014c82e>] remove_vma+0x1c/0x36
 [<c014c912>] exit_mmap+0xca/0xe1
 [<c011eedc>] mmput+0x1d/0x75
 [<c0123688>] do_exit+0x1be/0x68a
 [<c0123bc0>] sys_exit_group+0x0/0xd
 [<c0103d12>] sysenter_past_esp+0x5f/0x85
 =======================
Code: 00 52 41 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 fa 00 00 00 ea 05 00 00 7f 00 00 00 34 a5 96 <c1> 8c
00 fd f3 a5 2b 09 00 d1 d8 32 c0 00 c0 55 f2 00 a0 96 c1 EIP:
[<f255c08f>] 0xf255c08f SS:ESP 0068:f3631ea8 Kernel panic - not
syncing: Fatal exception in interrupt
----------------------------------------------------------------------

Test box is treated with mirrored traffic that is routed by production
linux router (with non-smp kernel). It's usual traffic generated by
broadband clients. Some of the characteristics:

bandwidth used: ~40/40 Mbit (up/down)
pps: ~15k

number of clients: ~550

dump of packet sizes:
Packet size  | Count
1 to  75:    186501
76 to  150:    14145
151 to  225:    3285
226 to  300:    2088
301 to  375:    3632
376 to  450:    2097
451 to  525:    1513
526 to  600:    3069
601 to  675:    20081
676 to  750:    1294
751 to  825:    1189
826 to  900:      885
901 to  975:    2207
976 to 1050:    1333
1051 to 1125:    1192
1201 to 1275:    3036
1276 to 1350:    3709
1351 to 1425:    3453
1426 to 1500+: 185318

protocol breakdown:
most of the traffic is IPv4, some UDP and a little bit of ICMP

Box is connected to switch by a
802.1q trunk. Each of vlan interfaces has egress shaping on it +
dedicated ifb device attached to ingress qdisc and ingress shaping on
ifb device.
Comment 1 Krzysztof Mościcki 2007-11-07 12:11:36 UTC
Created attachment 13442 [details]
kernel 2.6.23.1 - config
Comment 2 Marek Kierdelewicz 2007-12-24 10:29:54 UTC
Other people reported this bug on various forums/mailing lists. Examples:

http://forums.gentoo.org/viewtopic-t-458545.html
and
http://lists.openwall.net/netdev/2007/12/05/43

It seems to be SMP & ifb related
Comment 3 Jarek Poplawski 2007-12-27 10:32:02 UTC
This looks very similar too:
http://bugzilla.kernel.org/show_bug.cgi?id=9632

Marek, I think, it should be very helpful if you could repeat this test with
this new Oleg Nesterov's patch, which could tell us which timer breaks here.
Comment 4 Jarek Poplawski 2007-12-27 10:41:20 UTC
...Probably I should rather say Krzysztof? Or both?! Sorry guys!
Comment 5 Marek Kierdelewicz 2007-12-27 16:42:27 UTC
Thanks Jarek. Krzysztof and I are both work at the same company and we're at the same bottom of a pond with this bug metaphorically speaking.

Thanks to Jamal Hadi Salim I'm aware of the bug #9632 since this morning. I've got testing environment that allows to reproduce kernel panic in 5-15 minutes. I gave the patch a try but with no positive outcome I'm afraid :(.

Anyway bug #9632 may be a different thing. I've noticed my problems only with SMP+IFB+ingress in use. Badalian's example lets me think that he's not using IFB or ingress. Kernel panic message is different too. Badalian's message starts with "kernel BUG at kernel/timer.c:606!"
and in this but there's:
"PANIC: double fault, gdt at c1805000 [255 bytes]"
or
"general protection fault"

Jamal has given me some pointers of how to conduct further testing to narrow down the problem and he promised more help.
Comment 6 Jarek Poplawski 2007-12-27 22:51:11 UTC
> Thanks to Jamal Hadi Salim I'm aware of the bug #9632 since this morning.
> I've
> got testing environment that allows to reproduce kernel panic in 5-15
> minutes.
> I gave the patch a try but with no positive outcome I'm afraid :(.

Since this bug could be quite old I don't think there is any reason
to hurry. But it would be nice to fix this some day, of course.

> Anyway bug #9632 may be a different thing. I've noticed my problems only with
> SMP+IFB+ingress in use. Badalian's example lets me think that he's not using
> IFB or ingress. Kernel panic message is different too. Badalian's message
> starts with "kernel BUG at kernel/timer.c:606!"
> and in this but there's:
> "PANIC: double fault, gdt at c1805000 [255 bytes]"
> or
> "general protection fault"

I don't claim it's the same bug, but since it breaks while timers
are called, this Oleg's new patch could be very useful. So, if e.g.
timer_list structure is used after kfree (e.g. when it's not
deleted properly and the timer 'rearms' itself), it can break in
a very unpredictable way, and it could be some different timer, as
well. (BTW, it seems Badalian prefers 'Hungarian' way of signing,
but it's a surname...)

> Jamal has given me some pointers of how to conduct further testing to narrow
> down the problem and he promised more help.

Very nice! I hope you'll mention here any new observations. Thanks.
Comment 7 Jarek Poplawski 2007-12-27 23:01:38 UTC
> I gave the patch a try but with no positive outcome I'm afraid :(.

Hmm... I didn't read this with attention: does 'no positive outcome'
mean that with this new patch this still oopses exactly the same?
Comment 8 Marek Kierdelewicz 2007-12-30 04:20:35 UTC
> Hmm... I didn't read this with attention: does 'no positive outcome'
> mean that with this new patch this still oopses exactly the same?

Yup... oopses the same way.
Comment 9 Jarek Poplawski 2007-12-30 13:52:17 UTC
Created attachment 14231 [details]
Print timer running function in panic

Marek or Krzysztof, maybe you could try this test with this patch yet?
This could tell us the name of a timer function running during this panic. Thanks!
Comment 10 Jarek Poplawski 2007-12-31 08:44:28 UTC
Created attachment 14239 [details]
Print in panic plus verify timer function address

Marek and Krzysztof! I wish you great New Year's Party!
(But, if there were any problems ...here is a bit completed testing patch!)
Cheers, Jarek P.
Comment 11 Alan 2008-09-26 05:26:44 UTC
Closing out old stale bugs