Most recent kernel where this bug did not occur: Distribution: gentoo Hardware Environment: see below Software Environment: see below Problem Description: My company's (ISP) bussines model requires dynamic resizing of the client queues. It's achieved by regenerating shaping rules and loading then using batch mode of a tc binary. On production systems it's done once every 1 or 2 minutes. Unfortunately this causes smp kernels to panic. Non-smp kernels don't have such problems. Bug is around a long time. I first noticed it after migrating to shaping configs that use IFB, it might have been 2.6.18 "era". Steps to reproduce: I'm feeding the machine with production traffic by means of port mirroring. Test machine has the same config as production one (including mac addresses), so it tries to route the incoming traffic. Tested kernels were 2.6.23.1 and 2.6.20.6. Both panicked if compiled with SMP support and work stable otherwise. Problem occurs only with cyclic "shaping restarts". For the test, reload operation using "tc -b ..." was executed in an infinite loop. Box's CPU usage was approximately 15%. Panics occur with few hours - one day intervals. Below I attach the panic message captured via serial console: ---------------------------------------------------------------------- printk: 63 messages suppressed. dst cache overflow SMP Modules linked in: ipt_LOG xt_hashlimit ipt_MASQUERADE ip_set_macipmap ip_set_ipmap xt_state w83627hf hwmon_vid eeprom ifb ipt_SET ipt_set ip_set ipip tunnel4 ip_gre e1000 i2c_i801 i2c_core CPU: 1 EIP: 0060:[<f255c08f>] Not tainted VLI EFLAGS: 00010202 (2.6.23.1-smp #2) EIP is at 0xf255c08f eax: c196a000 ebx: 00000100 ecx: ef2f408c edx: f3630000 esi: f0332029 edi: f255c08c ebp: 00000001 esp: f3631ea8 ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 Process tc (pid: 27695, ti=f3630000 task=f26a9560 task.ti=f3630000) Stack: c01280ad f26a9560 00000001 f3631eb4 c0106f57 ef2f408c f0e8c08c 00000031 c0495308 0000000a c0124eb6 00000046 00000000 f7657740 f3630000 c0124f4c c180f120 c0114e62 00000000 00000000 f7bfd224 f74ed95c c01047e0 f7bfd224 Call Trace: [<c01280ad>] run_timer_softirq+0xf5/0x154 [<c0106f57>] profile_pc+0x21/0x4a [<c0124eb6>] __do_softirq+0x5d/0xc1 [<c0124f4c>] do_softirq+0x32/0x36 [<c0114e62>] smp_apic_timer_interrupt+0x74/0x80 [<c01047e0>] apic_timer_interrupt+0x28/0x30 [<c014c82e>] remove_vma+0x1c/0x36 [<c014c912>] exit_mmap+0xca/0xe1 [<c011eedc>] mmput+0x1d/0x75 [<c0123688>] do_exit+0x1be/0x68a [<c0123bc0>] sys_exit_group+0x0/0xd [<c0103d12>] sysenter_past_esp+0x5f/0x85 ======================= Code: 00 52 41 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa 00 00 00 ea 05 00 00 7f 00 00 00 34 a5 96 <c1> 8c 00 fd f3 a5 2b 09 00 d1 d8 32 c0 00 c0 55 f2 00 a0 96 c1 EIP: [<f255c08f>] 0xf255c08f SS:ESP 0068:f3631ea8 Kernel panic - not syncing: Fatal exception in interrupt ---------------------------------------------------------------------- Test box is treated with mirrored traffic that is routed by production linux router (with non-smp kernel). It's usual traffic generated by broadband clients. Some of the characteristics: bandwidth used: ~40/40 Mbit (up/down) pps: ~15k number of clients: ~550 dump of packet sizes: Packet size | Count 1 to 75: 186501 76 to 150: 14145 151 to 225: 3285 226 to 300: 2088 301 to 375: 3632 376 to 450: 2097 451 to 525: 1513 526 to 600: 3069 601 to 675: 20081 676 to 750: 1294 751 to 825: 1189 826 to 900: 885 901 to 975: 2207 976 to 1050: 1333 1051 to 1125: 1192 1201 to 1275: 3036 1276 to 1350: 3709 1351 to 1425: 3453 1426 to 1500+: 185318 protocol breakdown: most of the traffic is IPv4, some UDP and a little bit of ICMP Box is connected to switch by a 802.1q trunk. Each of vlan interfaces has egress shaping on it + dedicated ifb device attached to ingress qdisc and ingress shaping on ifb device.
Created attachment 13442 [details] kernel 2.6.23.1 - config
Other people reported this bug on various forums/mailing lists. Examples: http://forums.gentoo.org/viewtopic-t-458545.html and http://lists.openwall.net/netdev/2007/12/05/43 It seems to be SMP & ifb related
This looks very similar too: http://bugzilla.kernel.org/show_bug.cgi?id=9632 Marek, I think, it should be very helpful if you could repeat this test with this new Oleg Nesterov's patch, which could tell us which timer breaks here.
...Probably I should rather say Krzysztof? Or both?! Sorry guys!
Thanks Jarek. Krzysztof and I are both work at the same company and we're at the same bottom of a pond with this bug metaphorically speaking. Thanks to Jamal Hadi Salim I'm aware of the bug #9632 since this morning. I've got testing environment that allows to reproduce kernel panic in 5-15 minutes. I gave the patch a try but with no positive outcome I'm afraid :(. Anyway bug #9632 may be a different thing. I've noticed my problems only with SMP+IFB+ingress in use. Badalian's example lets me think that he's not using IFB or ingress. Kernel panic message is different too. Badalian's message starts with "kernel BUG at kernel/timer.c:606!" and in this but there's: "PANIC: double fault, gdt at c1805000 [255 bytes]" or "general protection fault" Jamal has given me some pointers of how to conduct further testing to narrow down the problem and he promised more help.
> Thanks to Jamal Hadi Salim I'm aware of the bug #9632 since this morning. > I've > got testing environment that allows to reproduce kernel panic in 5-15 > minutes. > I gave the patch a try but with no positive outcome I'm afraid :(. Since this bug could be quite old I don't think there is any reason to hurry. But it would be nice to fix this some day, of course. > Anyway bug #9632 may be a different thing. I've noticed my problems only with > SMP+IFB+ingress in use. Badalian's example lets me think that he's not using > IFB or ingress. Kernel panic message is different too. Badalian's message > starts with "kernel BUG at kernel/timer.c:606!" > and in this but there's: > "PANIC: double fault, gdt at c1805000 [255 bytes]" > or > "general protection fault" I don't claim it's the same bug, but since it breaks while timers are called, this Oleg's new patch could be very useful. So, if e.g. timer_list structure is used after kfree (e.g. when it's not deleted properly and the timer 'rearms' itself), it can break in a very unpredictable way, and it could be some different timer, as well. (BTW, it seems Badalian prefers 'Hungarian' way of signing, but it's a surname...) > Jamal has given me some pointers of how to conduct further testing to narrow > down the problem and he promised more help. Very nice! I hope you'll mention here any new observations. Thanks.
> I gave the patch a try but with no positive outcome I'm afraid :(. Hmm... I didn't read this with attention: does 'no positive outcome' mean that with this new patch this still oopses exactly the same?
> Hmm... I didn't read this with attention: does 'no positive outcome' > mean that with this new patch this still oopses exactly the same? Yup... oopses the same way.
Created attachment 14231 [details] Print timer running function in panic Marek or Krzysztof, maybe you could try this test with this patch yet? This could tell us the name of a timer function running during this panic. Thanks!
Created attachment 14239 [details] Print in panic plus verify timer function address Marek and Krzysztof! I wish you great New Year's Party! (But, if there were any problems ...here is a bit completed testing patch!) Cheers, Jarek P.
Closing out old stale bugs