Bug 5248 - Rapid loading and unloading of iptables modules gives oops followed by panic
Summary: Rapid loading and unloading of iptables modules gives oops followed by panic
Status: REJECTED WILL_NOT_FIX
Alias: None
Product: Networking
Classification: Unclassified
Component: Netfilter/Iptables (show other bugs)
Hardware: i386 Linux
: P2 high
Assignee: Harald Welte
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-09-13 17:55 UTC by Greg Bailey
Modified: 2008-03-14 00:05 UTC (History)
2 users (show)

See Also:
Kernel Version: 2.4.33-pre1
Subsystem:
Regression: ---
Bisected commit-id:


Attachments
Oops observed. iptables module(s) can't be completely unloaded (752 bytes, text/plain)
2005-09-13 17:56 UTC, Greg Bailey
Details
Kernel panic (partial) when restart iptables (1.04 KB, text/plain)
2005-09-13 17:58 UTC, Greg Bailey
Details
.config file for 2.4.32-pre3 kernel (41.09 KB, text/plain)
2005-09-13 22:57 UTC, Greg Bailey
Details
Loaded modules when firewall is enabled (785 bytes, text/plain)
2005-09-13 22:58 UTC, Greg Bailey
Details
Oops followed by panic - test 1 (3.20 KB, text/plain)
2005-09-14 12:04 UTC, Greg Bailey
Details
Oops followed by panic - test 2 (3.09 KB, text/plain)
2005-09-14 12:06 UTC, Greg Bailey
Details
Oops followed by panic - test 3 (1.90 KB, text/plain)
2005-09-14 12:06 UTC, Greg Bailey
Details
Oops followed by panic - test 2 (processed with ksymoops) (7.29 KB, text/plain)
2005-09-14 15:13 UTC, Greg Bailey
Details
2.4.33-pre1: oops (1.65 KB, text/plain)
2006-02-21 09:57 UTC, Greg Bailey
Details
2.4.33-pre1: panic (6.47 KB, text/plain)
2006-02-21 09:58 UTC, Greg Bailey
Details

Description Greg Bailey 2005-09-13 17:55:31 UTC
Most recent kernel where this bug did not occur:  N/A
Distribution:  RHEL3
Hardware Environment:  Intel Server Board SE7501CW2 2 Xeon 2.4GHz, 2G memory
Software Environment:  Server
Problem Description:  iptables is continually restarted via "service iptables
restart".  After a few minutes, the kernel either gives an oops or panics.

Steps to reproduce:  Do the following simultaneously:

A:
Continuously restart iptables (which loads and unloads modules)
while :; do service iptables restart; done

B:  
Continually clear and repopulate /etc/sysconfig/iptables
while :; do > /etc/sysconfig/iptables; sleep 2; /opt/xxx/bin/fwConfig.py -i;
sleep 2; done

C: (optional)
Hammer repeatedly with nmap
(on remote machine):  while :; do nmap -sT <victim>; done

D: (optional)
Hammer with apache benchmark utility (ab)
(on remote machine):  ./ab -n 10000000 <victim>/setup

After a few minutes, the kernel gives an oops when the /etc/sysconfig/iptables
is repopulated.  At this point, lsmod shows iptables modules as (deleted), but
they can not be removed or re-installed.

Sometimes the kernel will panic instead.  I've repeated this failure with 2.4.31
+ 2.4.32pre3 patch, Red Hat Enterprise 3 kernel 2.4.21-32.0.1smp, and 2.4.31 +
low latency patches (lck patchset).

The "/opt/xxx/bin/fwConfig.py" script referred to above regenerates the
/etc/sysconfig/iptables file and attempts to restart the iptables "service".

Panic and oops attached.
Comment 1 Greg Bailey 2005-09-13 17:56:53 UTC
Created attachment 6000 [details]
Oops observed.  iptables module(s) can't be completely unloaded
Comment 2 Greg Bailey 2005-09-13 17:58:17 UTC
Created attachment 6001 [details]
Kernel panic (partial) when restart iptables

Kernel panic when continuously restarting iptables.  This is the portion I
could observe on the screen and was typed in.
Comment 3 Greg Bailey 2005-09-13 22:57:31 UTC
Created attachment 6003 [details]
.config file for 2.4.32-pre3 kernel

kernel .config used to build kernel.  Based (loosely) on RHEL3 kernel config.
Comment 4 Greg Bailey 2005-09-13 22:58:51 UTC
Created attachment 6004 [details]
Loaded modules when firewall is enabled
Comment 5 Greg Bailey 2005-09-13 23:17:13 UTC
The modules that are loaded and removed by the "service iptables restart" are:

ipt_LOG                 4152   1  (autoclean)
ipt_state               1208   4  (autoclean)
ip_conntrack           32720   0  (autoclean) [ipt_state]
iptable_filter          2408   1  (autoclean)
ip_tables              14904   3  [ipt_LOG ipt_state iptable_filter]

Just to clarify, I've seen at least 2 *different* results of running my
netfilter "torture test":

1) I get an oops, the system is still "usable", but I can't reload new iptables
rules because one of the netfilter modules is shown as "(deleted)" by lsmod. 
This led me through many netfilter mailing lists and patches to attempt to see
if this had been addressed previously (like ip_conntrack not being able to be
unloaded, etc.)

2) The kernel panics.  I can't do anything on the system if that happens.

Again, both results come from running the same stress test but they didn't occur
at the same time.  I'm not familiar enough (yet) with netfilter to know if these
results are 2 different manifestations of the same problem or different problems
altogether.

I'll see about attempting to capture the oops output; I'm assuming I'll need to
configure the serial console thing.  Will post more when I have it; at least
this is fairly reproducible (for me, at least).
Comment 6 Greg Bailey 2005-09-14 12:04:15 UTC
Created attachment 6020 [details]
Oops followed by panic - test 1

The panic seems to have 2 different outputs intermingled together.  Anyone know
of a way to separate these two?
Comment 7 Greg Bailey 2005-09-14 12:06:22 UTC
Created attachment 6021 [details]
Oops followed by panic - test 2
Comment 8 Greg Bailey 2005-09-14 12:06:53 UTC
Created attachment 6022 [details]
Oops followed by panic - test 3
Comment 9 Greg Bailey 2005-09-14 12:09:22 UTC
To clarify what I said in Comment #5, it seems that the oops (which renders
iptables unusable as I can't unload/reload iptables modules) always precedes the
panic(s).  I may have missed the oops in prior runs and only saw the subsequent
panic.

Does anyone know how to separate 2 OOPS messages from the kernel that get
intermingled together?  My last 3 attachements have mangled output because of
this...
Comment 10 Harald Welte 2005-09-14 13:10:11 UTC
you have to process the oopses by kssymoops, otherwise the backtraces are
useless.  Please refer to /usr/src/linux/Documentation/oops-tracing.txt

Comment 11 Greg Bailey 2005-09-14 15:13:35 UTC
Created attachment 6027 [details]
Oops followed by panic - test 2 (processed with ksymoops)

Can't tell if ksymoops properly deciphered the later oopses that were
interleaved or not...
Comment 12 Greg Bailey 2006-02-21 09:57:28 UTC
Created attachment 7430 [details]
2.4.33-pre1: oops

This bug can still be triggered in 2.4.33-pre1.
Comment 13 Greg Bailey 2006-02-21 09:58:03 UTC
Created attachment 7431 [details]
2.4.33-pre1: panic

Problem reproduced in 2.4.33-pre1.
Comment 14 Natalie Protasevich 2008-03-14 00:05:34 UTC
Please be advised this bugzilla is for active development tree only meaning 2.6.XX. 2.4 bugs are being considered as exceptions, only if there is a very compelling reason to provide a fix.
I am closing the bug. If anyone has something to plea please reopen.
Comment 15 Willy Tarreau 2008-03-14 01:40:42 UTC
I was not aware of this one. I tend to agree with Natalie here. Unless an
obvious fix is proposed, I don't want to try to debug this issue for this
fairly uncommon case and risk to cause stability issues for normal users.

Note You need to log in before you can comment on or make changes to this bug.