21512 – Kernel BUG: NULL pointer dereference at nf_nat_setup_info [nf_nat]

Bug 21512 - Kernel BUG: NULL pointer dereference at nf_nat_setup_info [nf_nat]

Summary: Kernel BUG: NULL pointer dereference at nf_nat_setup_info [nf_nat]

Status:	RESOLVED CODE_FIX

Alias:	None

Product:	Networking
Classification:	Unclassified
Component:	Netfilter/Iptables (show other bugs)
Hardware:	All Linux

Importance:	P1 normal
Assignee:	networking_netfilter-iptables@kernel-bugs.osdl.org

URL:
Keywords:

Depends on:
Blocks:

Reported:	2010-10-31 01:15 UTC by Marian Ivasiuk
Modified:	2012-08-14 11:31 UTC (History)
CC List:	2 users (show)

See Also:
Kernel Version:
Subsystem:
Regression:	No
Bisected commit-id:

Attachments
nf_nat.ko module from crash report (533.31 KB, application/octet-stream) 2010-11-03 11:26 UTC, Marian Ivasiuk	Details
Add an attachment (proposed patch, testcase, etc.)

Description Marian Ivasiuk 2010-10-31 01:15:56 UTC

After upgrade from CentOS 5 (2.6.18) to newer systems (from fc11 to rhel6b2) with TX multiqueue support my Internet gateway (forwarding, REDIRECT, SNAT, ACCOUNT, filtering and shaping of 2+ Gbit/s traffic) always crashed under load over 1 Gbit/s.


Hardware:

Supermicro X8DTU-F with two Xeon X5650 and dual-port Intel 82598EB 10-Gigabit AT CX4 Network Connection (ixgbe driver from e1000.sf.net).


System tuned for more aggresive conntrack timeouts:

net.netfilter.nf_conntrack_generic_timeout = 60
net.netfilter.nf_conntrack_icmp_timeout = 60
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 0
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 1800
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 20
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 10


Netconsole logs from last crashes on rhel6b2 kernel (2.6.32-44.2.el6) with two patches (ipset and ipt_ACCOUNT):

Oct 29 13:56:52 prime BUG: unable to handle kernel 
Oct 29 13:56:52 prime NULL pointer dereference
Oct 29 13:56:52 prime  at 00000036 
Oct 29 13:56:52 prime IP:
Oct 29 13:56:52 prime  [<f86c3e3b>] nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 29 13:56:52 prime *pdpt = 0000000000adf001 
Oct 29 13:56:52 prime *pde = 0000000000000000 
Oct 29 13:56:52 prime  
Oct 29 13:56:52 prime Oops: 0000 [#1] 
Oct 29 13:56:52 prime SMP 
Oct 29 13:56:52 prime  
Oct 29 13:56:52 prime last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/0000:09:00.0/net/eth0/broadcast 
Oct 29 13:56:52 prime Modules linked in:
Oct 29 13:56:52 prime  netconsole
Oct 29 13:56:52 prime  configfs
Oct 29 13:56:52 prime  nf_conntrack_netlink
Oct 29 13:56:52 prime  nfnetlink
Oct 29 13:56:52 prime  cls_u32
Oct 29 13:56:52 prime  cls_fw
Oct 29 13:56:52 prime  sch_sfq
Oct 29 13:56:52 prime  sch_htb
Oct 29 13:56:52 prime  ipv6
Oct 29 13:56:52 prime  xt_statistic
Oct 29 13:56:52 prime  ts_kmp
Oct 29 13:56:52 prime  ipt_ULOG
Oct 29 13:56:52 prime  xt_comment
Oct 29 13:56:52 prime  ipt_ACCOUNT
Oct 29 13:56:52 prime  ipt_SET
Oct 29 13:56:52 prime  xt_string
Oct 29 13:56:52 prime  xt_hashlimit
Oct 29 13:56:52 prime  xt_multiport
Oct 29 13:56:52 prime  xt_connlimit
Oct 29 13:56:52 prime  iptable_nat
Oct 29 13:56:52 prime  nf_nat
Oct 29 13:56:52 prime  xt_DSCP
Oct 29 13:56:52 prime  ipt_set
Oct 29 13:56:52 prime  xt_mark
Oct 29 13:56:52 prime  xt_MARK
Oct 29 13:56:52 prime  iptable_mangle
Oct 29 13:56:52 prime  ip_set_portmap
Oct 29 13:56:52 prime  ip_set_iptree
Oct 29 13:56:52 prime  ip_set_iphash
Oct 29 13:56:52 prime  ip_set_ipmap
Oct 29 13:56:52 prime  ip_set_nethash
Oct 29 13:56:52 prime  ip_set
Oct 29 13:56:52 prime  dm_mirror
Oct 29 13:56:52 prime  dm_region_hash
Oct 29 13:56:52 prime  dm_log
Oct 29 13:56:52 prime  i2c_i801
Oct 29 13:56:52 prime  i2c_core
Oct 29 13:56:52 prime  sg
Oct 29 13:56:52 prime  iTCO_wdt
Oct 29 13:56:52 prime  iTCO_vendor_support
Oct 29 13:56:52 prime  ioatdma
Oct 29 13:56:52 prime  i7core_edac
Oct 29 13:56:52 prime  edac_core
Oct 29 13:56:52 prime  ixgbe
Oct 29 13:56:52 prime (U)
Oct 29 13:56:52 prime  mdio
Oct 29 13:56:52 prime  igb
Oct 29 13:56:52 prime  dca
Oct 29 13:56:52 prime  ext4
Oct 29 13:56:52 prime  mbcache
Oct 29 13:56:52 prime  jbd2
Oct 29 13:56:52 prime  sd_mod
Oct 29 13:56:52 prime  crc_t10dif
Oct 29 13:56:52 prime  pata_acpi
Oct 29 13:56:52 prime  ata_generic
Oct 29 13:56:52 prime ESI: e21701fc EDI: dfbc805c EBP: f7105c9c ESP: f7105be4 
Oct 29 13:56:52 prime  [<c076cd8e>] ? net_rx_action+0xde/0x1c0 
Oct 29 13:56:52 prime  [<c0456add>] ? do_softirq+0x3d/0x50 
Oct 29 13:56:52 prime  [<c0408784>] ? cpu_idle+0x94/0xd0 
Oct 29 13:56:52 prime 10 
Oct 29 13:56:52 prime  
Oct 29 13:56:52 prime ---[ end trace ad775db16e5047b0 ]--- 
Oct 29 13:56:52 prime Kernel panic - not syncing: Fatal exception in interrupt 
Oct 29 13:56:52 prime Pid: 0, comm: swapper Tainted: G      D    2.6.32-44.2.el6.1.iva.i686 #1 
Oct 29 13:56:52 prime Call Trace: 
Oct 29 13:56:52 prime  [<c0819a28>] ? oops_end+0xc8/0xd0 
Oct 29 13:56:52 prime  [<c081ad20>] ? do_page_fault+0x0/0x480 
Oct 29 13:56:52 prime  [<c081b0c8>] ? do_page_fault+0x3a8/0x480 
Oct 29 13:56:52 prime  [<c05e1690>] ? rb_insert_color+0x80/0x100 
Oct 29 13:56:52 prime  [<c045f667>] ? lock_timer_base+0x27/0x50 
Oct 29 13:56:52 prime  [<c0818dfb>] ? error_code+0x73/0x78 
Oct 29 13:56:52 prime  [<f86d313e>] ? alloc_null_binding+0x4e/0xa0 [iptable_nat] 
Oct 29 13:56:52 prime  [<f86d3696>] ? nf_nat_out+0x66/0xe0 [iptable_nat] 
Oct 29 13:56:52 prime  [<c078c406>] ? nf_iterate+0x76/0x90 
Oct 29 13:56:52 prime  [<c07a1f20>] ? ip_finish_output+0x0/0x280 
Oct 29 13:56:52 prime last message repeated 2 times
Oct 29 13:56:52 prime  [<c079cb03>] ? ip_rcv_finish+0xf3/0x390 
Oct 29 13:56:52 prime  [<c076c451>] ? netif_receive_skb+0x301/0x560 
Oct 29 13:56:52 prime  [<c04b0661>] ? move_native_irq+0x11/0x50 
Oct 29 13:56:52 prime  [<c076cd8e>] ? net_rx_action+0xde/0x1c0 
Oct 29 13:56:52 prime  [<c04b0661>] ? move_native_irq+0x11/0x50 
Oct 29 13:56:52 prime  [<c040b6a0>] ? do_IRQ+0x50/0xc0 
Oct 29 13:56:52 prime  [<c0409ff0>] ? common_interrupt+0x30/0x38 

Oct 29 21:23:30 prime BUG: unable to handle kernel 
Oct 29 21:23:30 prime NULL pointer dereference
Oct 29 21:23:30 prime  at 00000036 
Oct 29 21:23:30 prime IP:
Oct 29 21:23:30 prime  [<f86c3e3b>] nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 29 21:23:30 prime *pdpt = 0000000000adf001 
Oct 29 21:23:30 prime *pde = 0000000000000000 
Oct 29 21:23:30 prime  
Oct 29 21:23:30 prime Oops: 0000 [#1] 
Oct 29 21:23:30 prime SMP 
Oct 29 21:23:30 prime  
Oct 29 21:23:30 prime last sysfs file: /sys/module/nfnetlink/initstate 
Oct 29 21:23:30 prime Modules linked in:
Oct 29 21:23:30 prime  nf_conntrack_netlink
Oct 29 21:23:30 prime  nfnetlink
Oct 29 21:23:30 prime  cls_u32
Oct 29 21:23:30 prime  cls_fw
Oct 29 21:23:30 prime  sch_sfq
Oct 29 21:23:30 prime  sch_htb
Oct 29 21:23:30 prime  netconsole
Oct 29 21:23:30 prime  configfs
Oct 29 21:23:30 prime  ipv6
Oct 29 21:23:30 prime  xt_statistic
Oct 29 21:23:30 prime  ts_kmp
Oct 29 21:23:30 prime  ipt_ULOG
Oct 29 21:23:30 prime  xt_comment
Oct 29 21:23:30 prime  ipt_ACCOUNT
Oct 29 21:23:30 prime  ipt_SET
Oct 29 21:23:30 prime  xt_connlimit
Oct 29 21:23:30 prime  xt_string
Oct 29 21:23:30 prime  xt_hashlimit
Oct 29 21:23:30 prime  xt_multiport
Oct 29 21:23:30 prime  iptable_nat
Oct 29 21:23:30 prime  nf_nat
Oct 29 21:23:30 prime  xt_DSCP
Oct 29 21:23:30 prime  ipt_set
Oct 29 21:23:30 prime  xt_mark
Oct 29 21:23:30 prime  xt_MARK
Oct 29 21:23:30 prime  iptable_mangle
Oct 29 21:23:30 prime  ip_set_portmap
Oct 29 21:23:30 prime  ip_set_iptree
Oct 29 21:23:30 prime  ip_set_iphash
Oct 29 21:23:30 prime  ip_set_ipmap
Oct 29 21:23:30 prime  ip_set_nethash
Oct 29 21:23:30 prime  ip_set
Oct 29 21:23:30 prime  dm_mirror
Oct 29 21:23:30 prime  dm_region_hash
Oct 29 21:23:30 prime  dm_log
Oct 29 21:23:30 prime  i2c_i801
Oct 29 21:23:30 prime  i2c_core
Oct 29 21:23:30 prime  sg
Oct 29 21:23:30 prime  iTCO_wdt
Oct 29 21:23:30 prime  iTCO_vendor_support
Oct 29 21:23:30 prime  ioatdma
Oct 29 21:23:30 prime  i7core_edac
Oct 29 21:23:30 prime  edac_core
Oct 29 21:23:30 prime  ixgbe
Oct 29 21:23:30 prime (U)
Oct 29 21:23:30 prime  mdio
Oct 29 21:23:30 prime  igb
Oct 29 21:23:30 prime  dca
Oct 29 21:23:30 prime  ext4
Oct 29 21:23:30 prime  mbcache
Oct 29 21:23:30 prime  jbd2
Oct 29 21:23:30 prime  sd_mod
Oct 29 21:23:30 prime  crc_t10dif
Oct 29 21:23:30 prime  pata_acpi
Oct 29 21:23:30 prime  ata_generic
Oct 29 21:23:30 prime  ata_piix
Oct 29 21:23:30 prime  dm_mod
Oct 29 21:23:30 prime  [last unloaded: scsi_wait_scan]
Oct 29 21:23:30 prime  
Oct 29 21:23:30 prime EIP is at nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 29 21:23:30 prime  SS:ESP 0068:f7139bf8 
Oct 29 21:23:30 prime ---[ end trace 6b1ca7cebcce986a ]--- 
Oct 29 21:23:30 prime Kernel panic - not syncing: Fatal exception in interrupt 
Oct 29 21:23:30 prime  [<c081636e>] ? panic+0x42/0xed 
Oct 29 21:23:30 prime  [<c081ad20>] ? do_page_fault+0x0/0x480 
Oct 29 21:23:30 prime  [<c081b0c8>] ? do_page_fault+0x3a8/0x480 
Oct 29 21:23:30 prime  [<c0425804>] ? smp_apic_timer_interrupt+0x54/0x90 
Oct 29 21:23:30 prime  [<f86c3e3b>] ? nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 29 21:23:30 prime  [<f86d340c>] ? nf_nat_fn+0xec/0x210 [iptable_nat] 
Oct 29 21:23:30 prime  [<c078c406>] ? nf_iterate+0x76/0x90 
Oct 29 21:23:30 prime  [<c07a1f20>] ? ip_finish_output+0x0/0x280 
Oct 29 21:23:30 prime  [<c07a222a>] ? ip_output+0x8a/0xb0 
Oct 29 21:23:30 prime  [<c079cb03>] ? ip_rcv_finish+0xf3/0x390 
Oct 29 21:23:30 prime  [<c076c451>] ? netif_receive_skb+0x301/0x560 
Oct 29 21:23:30 prime  [<f811df69>] ? ixgbe_clean_rxtx_many+0xf9/0x1c0 [ixgbe] 
Oct 29 21:23:30 prime  [<c0456add>] ? do_softirq+0x3d/0x50 
Oct 29 21:23:30 prime  [<c040a335>] ? apic_timer_interrupt+0x31/0x38 
Oct 29 21:23:30 prime  [<c0408784>] ? cpu_idle+0x94/0xd0 

Oct 30 15:16:33 prime BUG: unable to handle kernel 
Oct 30 15:16:33 prime NULL pointer dereference
Oct 30 15:16:33 prime  at 00000036 
Oct 30 15:16:33 prime IP:
Oct 30 15:16:33 prime  [<f86c3e3b>] nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 30 15:16:33 prime *pdpt = 0000000000adf001 
Oct 30 15:16:33 prime *pde = 0000000000000000 
Oct 30 15:16:33 prime  
Oct 30 15:16:33 prime Oops: 0000 [#1] 
Oct 30 15:16:33 prime SMP 
Oct 30 15:16:33 prime  
Oct 30 15:16:33 prime last sysfs file: /sys/module/nfnetlink/initstate 
Oct 30 15:16:33 prime Modules linked in:
Oct 30 15:16:33 prime  nf_conntrack_netlink
Oct 30 15:16:33 prime  nfnetlink
Oct 30 15:16:33 prime  cls_u32
Oct 30 15:16:33 prime  cls_fw
Oct 30 15:16:33 prime  sch_sfq
Oct 30 15:16:33 prime  sch_htb
Oct 30 15:16:33 prime  netconsole
Oct 30 15:16:33 prime  configfs
Oct 30 15:16:33 prime  ipv6
Oct 30 15:16:33 prime  xt_statistic
Oct 30 15:16:33 prime  ts_kmp
Oct 30 15:16:33 prime  ipt_ULOG
Oct 30 15:16:33 prime  xt_comment
Oct 30 15:16:33 prime  ipt_ACCOUNT
Oct 30 15:16:33 prime  ipt_SET
Oct 30 15:16:33 prime  xt_connlimit
Oct 30 15:16:33 prime  xt_string
Oct 30 15:16:33 prime  xt_hashlimit
Oct 30 15:16:33 prime  xt_multiport
Oct 30 15:16:33 prime  iptable_nat
Oct 30 15:16:33 prime  nf_nat
Oct 30 15:16:33 prime  xt_DSCP
Oct 30 15:16:33 prime  ipt_set
Oct 30 15:16:33 prime  xt_mark
Oct 30 15:16:33 prime  xt_MARK
Oct 30 15:16:33 prime  iptable_mangle
Oct 30 15:16:33 prime  ip_set_portmap
Oct 30 15:16:33 prime  ip_set_iptree
Oct 30 15:16:33 prime  ip_set_iphash
Oct 30 15:16:33 prime  ip_set_ipmap
Oct 30 15:16:33 prime  ip_set_nethash
Oct 30 15:16:33 prime  ip_set
Oct 30 15:16:33 prime  dm_mirror
Oct 30 15:16:33 prime  dm_region_hash
Oct 30 15:16:33 prime  dm_log
Oct 30 15:16:33 prime  i2c_i801
Oct 30 15:16:33 prime  i2c_core
Oct 30 15:16:33 prime  sg
Oct 30 15:16:33 prime  iTCO_wdt
Oct 30 15:16:33 prime  iTCO_vendor_support
Oct 30 15:16:33 prime  ioatdma
Oct 30 15:16:33 prime  i7core_edac
Oct 30 15:16:33 prime  edac_core
Oct 30 15:16:33 prime  ixgbe
Oct 30 15:16:33 prime (U)
Oct 30 15:16:33 prime  mdio
Oct 30 15:16:33 prime  igb
Oct 30 15:16:33 prime  dca
Oct 30 15:16:33 prime  ext4
Oct 30 15:16:33 prime  mbcache
Oct 30 15:16:33 prime  jbd2
Oct 30 15:16:33 prime  sd_mod
Oct 30 15:16:33 prime  crc_t10dif
Oct 30 15:16:33 prime  pata_acpi
Oct 30 15:16:33 prime  ata_generic
Oct 30 15:16:33 prime  
Oct 30 15:16:33 prime  [<c079ca10>] ? ip_rcv_finish+0x0/0x390 
Oct 30 15:16:33 prime  [<f811df69>] ? ixgbe_clean_rxtx_many+0xf9/0x1c0 [ixgbe] 
Oct 30 15:16:33 prime  [<c045697f>] ? __do_softirq+0x8f/0x1b0 
Oct 30 15:16:33 prime  [<c0456add>] ? do_softirq+0x3d/0x50 
Oct 30 15:16:33 prime  [<c040b6a0>] ? do_IRQ+0x50/0xc0 
Oct 30 15:16:33 prime  [<c041114d>] ? mwait_idle+0x4d/0x90 
Oct 30 15:16:33 prime  [<c0408784>] ? cpu_idle+0x94/0xd0 

Oct 30 19:09:26 prime BUG: unable to handle kernel 
Oct 30 19:09:26 prime NULL pointer dereference
Oct 30 19:09:26 prime  at 00000036 
Oct 30 19:09:26 prime IP:
Oct 30 19:09:26 prime  [<f86c3e3b>] nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 
Oct 30 19:09:26 prime *pdpt = 000000002def6001 
Oct 30 19:09:26 prime *pde = 000000033dfa8067 
Oct 30 19:09:26 prime  
Oct 30 19:09:26 prime Oops: 0000 [#1] 
Oct 30 19:09:26 prime SMP 
Oct 30 19:09:26 prime  
Oct 30 19:09:26 prime last sysfs file: /sys/module/nfnetlink/initstate 
Oct 30 19:09:26 prime Modules linked in:
Oct 30 19:09:26 prime  nf_conntrack_netlink
Oct 30 19:09:26 prime  nfnetlink
Oct 30 19:09:26 prime  cls_u32
Oct 30 19:09:26 prime  cls_fw
Oct 30 19:09:26 prime  sch_sfq
Oct 30 19:09:26 prime  sch_htb
Oct 30 19:09:26 prime  netconsole
Oct 30 19:09:26 prime  configfs
Oct 30 19:09:26 prime  ipv6
Oct 30 19:09:26 prime  xt_statistic
Oct 30 19:09:26 prime  ts_kmp
Oct 30 19:09:26 prime  ipt_ULOG
Oct 30 19:09:26 prime  xt_comment
Oct 30 19:09:26 prime  ipt_ACCOUNT
Oct 30 19:09:26 prime  ipt_SET
Oct 30 19:09:26 prime  xt_connlimit
Oct 30 19:09:26 prime  xt_string
Oct 30 19:09:26 prime  xt_hashlimit
Oct 30 19:09:26 prime  xt_multiport
Oct 30 19:09:26 prime  iptable_nat
Oct 30 19:09:26 prime  nf_nat
Oct 30 19:09:26 prime  xt_DSCP
Oct 30 19:09:26 prime  ipt_set
Oct 30 19:09:26 prime  xt_mark
Oct 30 19:09:26 prime  xt_MARK
Oct 30 19:09:26 prime  iptable_mangle
Oct 30 19:09:26 prime  ip_set_portmap
Oct 30 19:09:26 prime  ip_set_iptree
Oct 30 19:09:26 prime  ip_set_iphash
Oct 30 19:09:26 prime  ip_set_ipmap
Oct 30 19:09:26 prime  ip_set_nethash
Oct 30 19:09:26 prime  ip_set
Oct 30 19:09:26 prime  dm_mirror
Oct 30 19:09:26 prime  dm_region_hash
Oct 30 19:09:26 prime  dm_log
Oct 30 19:09:26 prime  i2c_i801
Oct 30 19:09:26 prime  i2c_core
Oct 30 19:09:26 prime  sg
Oct 30 19:09:26 prime  iTCO_wdt
Oct 30 19:09:26 prime  iTCO_vendor_support
Oct 30 19:09:26 prime  ioatdma
Oct 30 19:09:26 prime  i7core_edac
Oct 30 19:09:26 prime  edac_core
Oct 30 19:09:26 prime  ixgbe
Oct 30 19:09:26 prime (U)
Oct 30 19:09:26 prime  mdio
Oct 30 19:09:26 prime  igb
Oct 30 19:09:26 prime  dca
Oct 30 19:09:26 prime  ext4
Oct 30 19:09:26 prime  mbcache
Oct 30 19:09:26 prime  jbd2
Oct 30 19:09:26 prime  sd_mod
Oct 30 19:09:26 prime  crc_t10dif
Oct 30 19:09:26 prime  pata_acpi
Oct 30 19:09:26 prime  ata_generic
Oct 30 19:09:26 prime  ata_piix
Oct 30 19:09:26 prime ESI: e9279cfc EDI: da9c805c EBP: f7261bf8 ESP: f7261b40 
Oct 30 19:09:26 prime  [<c047edef>] ? tick_dev_program_event+0x6f/0xd0 
Oct 30 19:09:26 prime  [<c04b0661>] ? move_native_irq+0x11/0x50 
Oct 30 19:09:26 prime  [<c0425804>] ? smp_apic_timer_interrupt+0x54/0x90 
Oct 30 19:09:26 prime 00 
Oct 30 19:09:26 prime CR2: 0000000000000036 
Oct 30 19:09:26 prime ---[ end trace 0a3052827abea68e ]--- 
Oct 30 19:09:26 prime Kernel panic - not syncing: Fatal exception in interrupt 
Oct 30 19:09:26 prime  [<c0819a28>] ? oops_end+0xc8/0xd0 
Oct 30 19:09:26 prime  [<c081ad20>] ? do_page_fault+0x0/0x480 
Oct 30 19:09:26 prime  [<c0818dfb>] ? error_code+0x73/0x78 
Oct 30 19:09:26 prime  [<f903f122>] ? htb_enqueue+0x2e2/0x3a0 [sch_htb] 
Oct 30 19:09:26 prime  [<f8662326>] ? portmap_ktest+0x66/0x100 [ip_set_portmap] 
Oct 30 19:09:26 prime  [<f86d340c>] ? nf_nat_fn+0xec/0x210 [iptable_nat] 
Oct 30 19:09:26 prime  [<c078c406>] ? nf_iterate+0x76/0x90 
Oct 30 19:09:26 prime  [<c078c572>] ? nf_hook_slow+0x62/0xe0 
Oct 30 19:09:26 prime  [<c07a1f20>] ? ip_finish_output+0x0/0x280 
Oct 30 19:09:26 prime  [<c079ca10>] ? ip_rcv_finish+0x0/0x390 
Oct 30 19:09:26 prime  [<c076c451>] ? netif_receive_skb+0x301/0x560 
Oct 30 19:09:26 prime  [<f811df69>] ? ixgbe_clean_rxtx_many+0xf9/0x1c0 [ixgbe] 
Oct 30 19:09:26 prime  [<c04b0661>] ? move_native_irq+0x11/0x50 
Oct 30 19:09:26 prime  [<c0456c35>] ? irq_exit+0x65/0x70 
Oct 30 19:09:26 prime  [<c0409ff0>] ? common_interrupt+0x30/0x38 
Oct 30 19:09:26 prime  [<c0812681>] ? start_secondary+0x209/0x24e

Comment 1 Patrick McHardy 2010-11-03 07:24:41 UTC

On 31.10.2010 02:15, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=21512
> 
>            Summary: Kernel BUG: NULL pointer dereference at
>                     nf_nat_setup_info [nf_nat]
>            Product: Networking
>            Version: 2.5
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Netfilter/Iptables
>         AssignedTo: networking_netfilter-iptables@kernel-bugs.osdl.org
>         ReportedBy: ivasiuk@gmail.com
>         Regression: No
> 
> 
> After upgrade from CentOS 5 (2.6.18) to newer systems (from fc11 to rhel6b2)
> with TX multiqueue support my Internet gateway (forwarding, REDIRECT, SNAT,
> ACCOUNT, filtering and shaping of 2+ Gbit/s traffic) always crashed under
> load
> over 1 Gbit/s.
> 
> 
> Hardware:
> 
> Supermicro X8DTU-F with two Xeon X5650 and dual-port Intel 82598EB 10-Gigabit
> AT CX4 Network Connection (ixgbe driver from e1000.sf.net).
> 
> 
> System tuned for more aggresive conntrack timeouts:
> 
> net.netfilter.nf_conntrack_generic_timeout = 60
> net.netfilter.nf_conntrack_icmp_timeout = 60
> net.netfilter.nf_conntrack_tcp_be_liberal = 0
> net.netfilter.nf_conntrack_tcp_loose = 0
> net.netfilter.nf_conntrack_tcp_timeout_close = 10
> net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
> net.netfilter.nf_conntrack_tcp_timeout_established = 1800
> net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
> net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
> net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 30
> net.netfilter.nf_conntrack_tcp_timeout_time_wait = 20
> net.netfilter.nf_conntrack_udp_timeout = 10
> net.netfilter.nf_conntrack_udp_timeout_stream = 10
> 
> 
> Netconsole logs from last crashes on rhel6b2 kernel (2.6.32-44.2.el6) with
> two
> patches (ipset and ipt_ACCOUNT):
> 
> Oct 29 13:56:52 prime BUG: unable to handle kernel 
> Oct 29 13:56:52 prime NULL pointer dereference
> Oct 29 13:56:52 prime  at 00000036 
> Oct 29 13:56:52 prime IP:
> Oct 29 13:56:52 prime  [<f86c3e3b>] nf_nat_setup_info+0x5fb/0x7a0 [nf_nat] 

Doesn't ring a bell. Please send your nf_nat object file and (in case
those are used on RHEL) the corresponding debuginfo file.

Comment 2 Marian Ivasiuk 2010-11-03 11:26:24 UTC

Created attachment 35992 [details]
nf_nat.ko module from crash report

In archive attached original nf_nat.ko module from crash report and the same module debuginfo rebuilded with corresponding nf_nat.ko.debug file.

Comment 3 Marian Ivasiuk 2010-11-03 11:34:15 UTC

Kernel crashed with the similar crash report even without any rule in NAT table but loaded nf_nat and iptable_nat modules.

Comment 4 Marian Ivasiuk 2010-11-22 21:15:07 UTC

Patrick, is more information needed on this case?

Comment 5 Oleg Arkhangelsky 2011-03-04 07:20:51 UTC

Marian, could you please try the following patch proposed by Changli? I see that in your case bug is triggered more often than in mine. So then if patch is fine we will know it quickly.

http://www.spinics.net/lists/netfilter-devel/msg17406.html

Thank you.

Comment 6 Marian Ivasiuk 2011-03-09 12:21:11 UTC

> Marian, could you please try the following patch proposed by Changli?
Proposed patch installed (kernel-2.6.32-71.18.1.el6) and now being tested on two identical systems with previous hardware configuration.

Comment 7 Oleg Arkhangelsky 2011-03-26 18:58:41 UTC

Any update?

Comment 8 Marian Ivasiuk 2011-03-31 13:54:29 UTC

> Any update?
Problem was fixed. Two earlier mentioned systems working without crashes for 22 days under load up to 4 Gbit/s.

Comment 9 Oleg Arkhangelsky 2011-03-31 14:02:22 UTC

Thank you for testing!

Note You need to log in before you can comment on or make changes to this bug.