Most recent kernel where this bug did not occur: N/A Distribution: RHEL3 Hardware Environment: Intel Server Board SE7501CW2 2 Xeon 2.4GHz, 2G memory Software Environment: Server Problem Description: iptables is continually restarted via "service iptables restart". After a few minutes, the kernel either gives an oops or panics. Steps to reproduce: Do the following simultaneously: A: Continuously restart iptables (which loads and unloads modules) while :; do service iptables restart; done B: Continually clear and repopulate /etc/sysconfig/iptables while :; do > /etc/sysconfig/iptables; sleep 2; /opt/xxx/bin/fwConfig.py -i; sleep 2; done C: (optional) Hammer repeatedly with nmap (on remote machine): while :; do nmap -sT <victim>; done D: (optional) Hammer with apache benchmark utility (ab) (on remote machine): ./ab -n 10000000 <victim>/setup After a few minutes, the kernel gives an oops when the /etc/sysconfig/iptables is repopulated. At this point, lsmod shows iptables modules as (deleted), but they can not be removed or re-installed. Sometimes the kernel will panic instead. I've repeated this failure with 2.4.31 + 2.4.32pre3 patch, Red Hat Enterprise 3 kernel 2.4.21-32.0.1smp, and 2.4.31 + low latency patches (lck patchset). The "/opt/xxx/bin/fwConfig.py" script referred to above regenerates the /etc/sysconfig/iptables file and attempts to restart the iptables "service". Panic and oops attached.
Created attachment 6000 [details] Oops observed. iptables module(s) can't be completely unloaded
Created attachment 6001 [details] Kernel panic (partial) when restart iptables Kernel panic when continuously restarting iptables. This is the portion I could observe on the screen and was typed in.
Created attachment 6003 [details] .config file for 2.4.32-pre3 kernel kernel .config used to build kernel. Based (loosely) on RHEL3 kernel config.
Created attachment 6004 [details] Loaded modules when firewall is enabled
The modules that are loaded and removed by the "service iptables restart" are: ipt_LOG 4152 1 (autoclean) ipt_state 1208 4 (autoclean) ip_conntrack 32720 0 (autoclean) [ipt_state] iptable_filter 2408 1 (autoclean) ip_tables 14904 3 [ipt_LOG ipt_state iptable_filter] Just to clarify, I've seen at least 2 *different* results of running my netfilter "torture test": 1) I get an oops, the system is still "usable", but I can't reload new iptables rules because one of the netfilter modules is shown as "(deleted)" by lsmod. This led me through many netfilter mailing lists and patches to attempt to see if this had been addressed previously (like ip_conntrack not being able to be unloaded, etc.) 2) The kernel panics. I can't do anything on the system if that happens. Again, both results come from running the same stress test but they didn't occur at the same time. I'm not familiar enough (yet) with netfilter to know if these results are 2 different manifestations of the same problem or different problems altogether. I'll see about attempting to capture the oops output; I'm assuming I'll need to configure the serial console thing. Will post more when I have it; at least this is fairly reproducible (for me, at least).
Created attachment 6020 [details] Oops followed by panic - test 1 The panic seems to have 2 different outputs intermingled together. Anyone know of a way to separate these two?
Created attachment 6021 [details] Oops followed by panic - test 2
Created attachment 6022 [details] Oops followed by panic - test 3
To clarify what I said in Comment #5, it seems that the oops (which renders iptables unusable as I can't unload/reload iptables modules) always precedes the panic(s). I may have missed the oops in prior runs and only saw the subsequent panic. Does anyone know how to separate 2 OOPS messages from the kernel that get intermingled together? My last 3 attachements have mangled output because of this...
you have to process the oopses by kssymoops, otherwise the backtraces are useless. Please refer to /usr/src/linux/Documentation/oops-tracing.txt
Created attachment 6027 [details] Oops followed by panic - test 2 (processed with ksymoops) Can't tell if ksymoops properly deciphered the later oopses that were interleaved or not...
Created attachment 7430 [details] 2.4.33-pre1: oops This bug can still be triggered in 2.4.33-pre1.
Created attachment 7431 [details] 2.4.33-pre1: panic Problem reproduced in 2.4.33-pre1.
Please be advised this bugzilla is for active development tree only meaning 2.6.XX. 2.4 bugs are being considered as exceptions, only if there is a very compelling reason to provide a fix. I am closing the bug. If anyone has something to plea please reopen.
I was not aware of this one. I tend to agree with Natalie here. Unless an obvious fix is proposed, I don't want to try to debug this issue for this fairly uncommon case and risk to cause stability issues for normal users.