Bug 65191
| Summary: | BUG in nf_nat_cleanup_conntrack | ||
|---|---|---|---|
| Product: | Networking | Reporter: | Samu Kallio (samu.kallio) |
| Component: | Netfilter/Iptables | Assignee: | networking_netfilter-iptables (networking_netfilter-iptables) |
| Status: | NEW --- | ||
| Severity: | normal | CC: | alan, dushistov, fw, gdm85, rodrigo.vaz, scottm, steve, szg00000, wwadge |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 3.10.17 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: | Don't touch nat_bysource hash chain when netns is going away, preventing crash. | ||
|
Description
Samu Kallio
2013-11-19 14:59:40 UTC
Can you test 3.12 on that setup and see if it is then stable ? I'll try to get a reliable test case going in the next few days. This issue is occurring on production servers, so I'm reluctant to upgrade away from long term stable there. I think I'm getting the same problem (this in an LXC environment) on a development server so feel free to ask for more info.
This is on a 3.13.1 kernel and triggers on UP as well as SMP fairly consistently.
KERNEL: /home/wwadge/linux-3.13.1/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 1
DATE: Sat Feb 1 14:34:00 2014
UPTIME: 21:15:10
LOAD AVERAGE: 0.10, 0.09, 0.06
TASKS: 498
NODENAME: cds-dws-centos
RELEASE: 3.13.1
VERSION: #3 SMP Fri Jan 31 16:51:22 GMT 2014
MACHINE: x86_64 (2660 Mhz)
MEMORY: 2 GB
PANIC:
WARNING: log buf data structure(s) have changed
""
PID: 11536
COMMAND: "kworker/u2:1"
TASK: ffff880037de12d0 [THREAD_INFO: ffff880000048000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 11536 TASK: ffff880037de12d0 CPU: 0 COMMAND: "kworker/u2:1"
#0 [ffff880000049850] machine_kexec at ffffffff8104ca40
#1 [ffff8800000498c0] crash_kexec at ffffffff810e26e8
#2 [ffff880000049990] oops_end at ffffffff81618258
#3 [ffff8800000499c0] no_context at ffffffff81056a1e
#4 [ffff880000049a10] __bad_area_nosemaphore at ffffffff81056c1d
#5 [ffff880000049a60] bad_area_nosemaphore at ffffffff81056d33
#6 [ffff880000049a70] __do_page_fault at ffffffff8161b046
#7 [ffff880000049b90] do_page_fault at ffffffff8161b26e
#8 [ffff880000049ba0] page_fault at ffffffff81617688
[exception RIP: nf_nat_cleanup_conntrack+70]
RIP: ffffffffa0331236 RSP: ffff880000049c58 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8800632ebe08 RCX: ffff8800566e2c18
RDX: ffffc90011b9e288 RSI: 0000000000000006 RDI: ffffffffa03343e8
RBP: ffff880000049c68 R8: dead000000200200 R9: dead000000200200
R10: dead000000200200 R11: ffffffc000000030 R12: ffff8800632ebd81
R13: ffff8800566e2c18 R14: ffff880000049d14 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#9 [ffff880000049c70] __nf_ct_ext_destroy at ffffffffa023a2b5 [nf_conntrack]
#10 [ffff880000049ca0] nf_conntrack_free at ffffffffa0231fdf [nf_conntrack]
#11 [ffff880000049cc0] destroy_conntrack at ffffffffa0232ee2 [nf_conntrack]
#12 [ffff880000049ce0] nf_conntrack_destroy at ffffffff8157d787
#13 [ffff880000049cf0] nf_ct_iterate_cleanup at ffffffffa02323f8 [nf_conntrack]
#14 [ffff880000049d50] nf_ct_l3proto_pernet_unregister at ffffffffa02374c9 [nf_conntrack]
#15 [ffff880000049d70] ipv4_net_exit at ffffffffa0250e2d [nf_conntrack_ipv4]
#16 [ffff880000049d90] ops_exit_list at ffffffff81546269
#17 [ffff880000049dc0] cleanup_net at ffffffff81546843
#18 [ffff880000049e00] process_one_work at ffffffff8108468c
#19 [ffff880000049e50] worker_thread at ffffffff81085ae3
#20 [ffff880000049ec0] kthread at ffffffff8108b73e
#21 [ffff880000049f50] ret_from_fork at ffffffff8161f9fc
----
crash> net
NET_DEVICE NAME IP ADDRESS(ES)
ffff88007ca9a000 lo 127.0.0.1
ffff88007b746000 eth0 10.20.70.9
ffff880075730000 virbr0 192.168.122.1
ffff88007c0dc000 virbr0-nic
ffff880075760000 docker0 172.17.42.1
ffff88006e5ea000 vethQBjNiA
ffff88006e42b000 vethKx1UUw
ffff88003ae94000 vethPPPQ5A
ffff88006334e000 veth9Lc57z
ffff8800566c0000 vethdlEGPB
ffff88003e8d2000 vethoU41sj
---
I can provide crashlog + .config too if need be.
On Sat, Feb 01, 2014 at 04:54:21PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=65191 > > Wallace Wadge <wwadge@gmail.com> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |wwadge@gmail.com > > --- Comment #3 from Wallace Wadge <wwadge@gmail.com> --- > I think I'm getting the same problem (this in an LXC environment) on a > development server so feel free to ask for more info. > > This is on a 3.13.1 kernel and triggers on UP as well as SMP fairly > consistently. Could you turn on CONFIG_NETFILTER_DEBUG and see if the assertion in nf_nat_cleanup_conntrack() hits please? Enabled but don't see asserts. I upgraded my crash utility so I got a little more info: --- [ 1515.714065] BUG: unable to handle kernel paging request at ffffc90011b95fc8 [ 1515.714118] IP: [<ffffffffa033623e>] nf_nat_cleanup_conntrack+0x4e/0x90 [nf_nat] [ 1515.714167] PGD 7f851067 PUD 7f852067 PMD 74426067 PTE 0 [ 1515.714208] Oops: 0002 [#1] SMP [ 1515.714240] Modules linked in: veth xt_nat xt_addrtype xt_conntrack ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_CHECKSUM iptable_mangle tun bridge stp llc dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ip6_tables ppdev vmw_balloon pcspkr parport_pc parport e1000 sg vmw_vmci i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif crct10dif_common sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix floppy vmwgfx ttm dm_mirror dm_region_hash dm_log dm_mod [ 1515.714656] CPU: 0 PID: 10628 Comm: kworker/u2:4 Not tainted 3.13.1 #4 [ 1515.714698] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 1515.714774] Workqueue: netns cleanup_net [ 1515.714806] task: ffff88001c4ea610 ti: ffff88001c4f0000 task.ti: ffff88001c4f0000 [ 1515.714855] RIP: 0010:[<ffffffffa033623e>] [<ffffffffa033623e>] nf_nat_cleanup_conntrack+0x4e/0x90 [nf_nat] [ 1515.714897] RSP: 0000:ffff88001c4f1c58 EFLAGS: 00010246 [ 1515.714917] RAX: 0000000000000000 RBX: ffff88003af1eb88 RCX: ffff880062792c18 [ 1515.714940] RDX: ffffc90011b95fc8 RSI: dead000000200200 RDI: ffffffffa03393e8 [ 1515.714962] RBP: ffff88001c4f1c68 R08: 0000000000000000 R09: dead000000200200 [ 1515.714986] R10: dead000000200200 R11: ffffffc000000030 R12: ffff88003af1eb01 [ 1515.715009] R13: ffff880062792c18 R14: ffff88001c4f1d14 R15: 0000000000000000 [ 1515.715033] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 1515.715060] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1515.715078] CR2: ffffc90011b95fc8 CR3: 000000001df89000 CR4: 00000000000007f0 [ 1515.715173] Stack: [ 1515.715178] ffff88001c4f1c78 ffffffffa0246c28 ffff88001c4f1c98 ffffffffa023f545 [ 1515.715208] ffff88001c4f1ce8 ffff880062792c18 ffff88000065e400 ffffffffa02583a0 [ 1515.715238] ffff88001c4f1cb8 ffffffffa023709f ffff880062792c18 ffff88000065e400 [ 1515.715266] Call Trace: [ 1515.715983] [<ffffffffa023f545>] __nf_ct_ext_destroy+0x45/0x70 [nf_conntrack] [ 1515.716664] [<ffffffffa023709f>] nf_conntrack_free+0x2f/0x70 [nf_conntrack] [ 1515.717345] [<ffffffffa02384ea>] destroy_conntrack+0xba/0x110 [nf_conntrack] [ 1515.718041] [<ffffffffa023c1e0>] ? nf_conntrack_helper_unregister+0xc0/0xc0 [nf_conntrack] [ 1515.718726] [<ffffffff8157d6c7>] nf_conntrack_destroy+0x17/0x30 [ 1515.719405] [<ffffffffa0238208>] nf_ct_iterate_cleanup+0x78/0xb0 [nf_conntrack] [ 1515.720087] [<ffffffffa023c759>] nf_ct_l3proto_pernet_unregister+0x39/0x80 [nf_conntrack] [ 1515.720765] [<ffffffffa0255ecd>] ipv4_net_exit+0x1d/0x60 [nf_conntrack_ipv4] [ 1515.721437] [<ffffffff81546269>] ops_exit_list+0x39/0x60 [ 1515.722104] [<ffffffff81546843>] cleanup_net+0x103/0x1b0 [ 1515.722752] [<ffffffff8108468c>] process_one_work+0x17c/0x420 [ 1515.723385] [<ffffffff81085ae3>] worker_thread+0x123/0x400 [ 1515.724000] [<ffffffff810859c0>] ? manage_workers+0x170/0x170 [ 1515.724600] [<ffffffff8108b73e>] kthread+0xce/0xf0 [ 1515.725180] [<ffffffff8108b670>] ? kthread_freezable_should_stop+0x70/0x70 [ 1515.725752] [<ffffffff8161fa7c>] ret_from_fork+0x7c/0xb0 [ 1515.726306] [<ffffffff8108b670>] ? kthread_freezable_should_stop+0x70/0x70 [ 1515.726860] Code: b6 db 48 8d 1c 18 48 8b 43 10 48 85 c0 74 3f 80 78 78 00 79 40 48 c7 c7 e8 93 33 a0 e8 1c 0d 2e e1 48 8b 03 48 8b 53 08 48 85 c0 <48> 89 02 74 04 48 89 50 08 48 b9 00 02 20 00 00 00 ad de 48 c7 [ 1515.728048] RIP [<ffffffffa033623e>] nf_nat_cleanup_conntrack+0x4e/0x90 [nf_nat] [ 1515.728622] RSP <ffff88001c4f1c58> [ 1515.729181] CR2: ffffc90011b95fc8 crash> --- And not sure if I got the address right in this debug attempt but here's some more info:
---
crash> bt
PID: 10628 TASK: ffff88001c4ea610 CPU: 0 COMMAND: "kworker/u2:4"
#0 [ffff88001c4f1850] machine_kexec at ffffffff8104ca40
#1 [ffff88001c4f18c0] crash_kexec at ffffffff810e26e8
#2 [ffff88001c4f1990] oops_end at ffffffff816182d8
#3 [ffff88001c4f19c0] no_context at ffffffff81056a1e
#4 [ffff88001c4f1a10] __bad_area_nosemaphore at ffffffff81056c1d
#5 [ffff88001c4f1a60] bad_area_nosemaphore at ffffffff81056d33
#6 [ffff88001c4f1a70] __do_page_fault at ffffffff8161b0c6
#7 [ffff88001c4f1b90] do_page_fault at ffffffff8161b2ee
#8 [ffff88001c4f1ba0] page_fault at ffffffff81617708
[exception RIP: nf_nat_cleanup_conntrack+78]
RIP: ffffffffa033623e RSP: ffff88001c4f1c58 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88003af1eb88 RCX: ffff880062792c18
RDX: ffffc90011b95fc8 RSI: dead000000200200 RDI: ffffffffa03393e8
RBP: ffff88001c4f1c68 R8: 0000000000000000 R9: dead000000200200
R10: dead000000200200 R11: ffffffc000000030 R12: ffff88003af1eb01
R13: ffff880062792c18 R14: ffff88001c4f1d14 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#9 [ffff88001c4f1c70] __nf_ct_ext_destroy at ffffffffa023f545 [nf_conntrack]
#10 [ffff88001c4f1ca0] nf_conntrack_free at ffffffffa023709f [nf_conntrack]
#11 [ffff88001c4f1cc0] destroy_conntrack at ffffffffa02384ea [nf_conntrack]
#12 [ffff88001c4f1ce0] nf_conntrack_destroy at ffffffff8157d6c7
#13 [ffff88001c4f1cf0] nf_ct_iterate_cleanup at ffffffffa0238208 [nf_conntrack]
#14 [ffff88001c4f1d50] nf_ct_l3proto_pernet_unregister at ffffffffa023c759 [nf_conntrack]
#15 [ffff88001c4f1d70] ipv4_net_exit at ffffffffa0255ecd [nf_conntrack_ipv4]
#16 [ffff88001c4f1d90] ops_exit_list at ffffffff81546269
#17 [ffff88001c4f1dc0] cleanup_net at ffffffff81546843
#18 [ffff88001c4f1e00] process_one_work at ffffffff8108468c
#19 [ffff88001c4f1e50] worker_thread at ffffffff81085ae3
#20 [ffff88001c4f1ec0] kthread at ffffffff8108b73e
#21 [ffff88001c4f1f50] ret_from_fork at ffffffff8161fa7c
crash> struct nf_conn ffff880062792c18
struct nf_conn {
ct_general = {
use = {
counter = 0
}
},
lock = {
{
rlock = {
raw_lock = {
{
head_tail = 445651600,
tickets = {
head = 6800,
tail = 6800
}
}
}
}
}
},
tuplehash = {{
hnnode = {
next = 0x80000003,
pprev = 0xdead000000200200
},
tuple = {
src = {
u3 = {
all = {335548844, 0, 0, 0},
ip = 335548844,
ip6 = {335548844, 0, 0, 0},
in = {
s_addr = 335548844
},
in6 = {
in6_u = {
u6_addr8 = "\254\021\000\024\000\000\000\000\000\000\000\000\000\000\000",
u6_addr16 = {4524, 5120, 0, 0, 0, 0, 0, 0},
u6_addr32 = {335548844, 0, 0, 0}
}
}
},
u = {
all = 15256,
tcp = {
port = 15256
},
udp = {
port = 15256
},
icmp = {
id = 15256
},
dccp = {
port = 15256
},
sctp = {
port = 15256
},
gre = {
key = 15256
}
},
l3num = 2
},
dst = {
u3 = {
all = {3054244874, 0, 0, 0},
ip = 3054244874,
ip6 = {3054244874, 0, 0, 0},
in = {
s_addr = 3054244874
},
in6 = {
in6_u = {
u6_addr8 = "\n\024\f\266\000\000\000\000\000\000\000\000\000\000\000",
u6_addr16 = {5130, 46604, 0, 0, 0, 0, 0, 0},
u6_addr32 = {3054244874, 0, 0, 0}
}
}
},
u = {
all = 37151,
tcp = {
port = 37151
},
udp = {
port = 37151
},
icmp = {
type = 31 '\037',
code = 145 '\221'
},
dccp = {
port = 37151
},
sctp = {
port = 37151
},
gre = {
key = 37151
}
},
protonum = 6 '\006',
dir = 0 '\000'
}
}
}, {
hnnode = {
next = 0x4727,
pprev = 0xdead000000200200
},
tuple = {
src = {
u3 = {
all = {3054244874, 0, 0, 0},
ip = 3054244874,
ip6 = {3054244874, 0, 0, 0},
in = {
s_addr = 3054244874
},
in6 = {
in6_u = {
u6_addr8 = "\n\024\f\266\000\000\000\000\000\000\000\000\000\000\000",
u6_addr16 = {5130, 46604, 0, 0, 0, 0, 0, 0},
u6_addr32 = {3054244874, 0, 0, 0}
}
}
},
u = {
all = 37151,
tcp = {
port = 37151
},
udp = {
port = 37151
},
icmp = {
id = 37151
},
dccp = {
port = 37151
},
sctp = {
port = 37151
},
gre = {
key = 37151
}
},
l3num = 2
},
dst = {
u3 = {
all = {335548844, 0, 0, 0},
ip = 335548844,
ip6 = {335548844, 0, 0, 0},
in = {
s_addr = 335548844
},
in6 = {
in6_u = {
u6_addr8 = "\254\021\000\024\000\000\000\000\000\000\000\000\000\000\000",
u6_addr16 = {4524, 5120, 0, 0, 0, 0, 0, 0},
u6_addr32 = {335548844, 0, 0, 0}
}
}
},
u = {
all = 15256,
tcp = {
port = 15256
},
udp = {
port = 15256
},
icmp = {
type = 152 '\230',
code = 59 ';'
},
dccp = {
port = 15256
},
sctp = {
port = 15256
},
gre = {
key = 15256
}
},
protonum = 6 '\006',
dir = 1 '\001'
}
}
}},
status = 910,
master = 0x0,
timeout = {
entry = {
next = 0x0,
prev = 0xdead000000200200
},
expires = 4296302640,
base = 0xffffffff81fb4f80 <boot_tvec_bases>,
function = 0xffffffffa0238410 <death_by_timeout>,
data = 18446612133966326808,
slack = -1,
start_pid = -1,
start_site = 0x0,
start_comm = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
},
mark = 0,
secmark = 0,
ext = 0xffff88003af1eb00,
ct_net = 0xffff88000065e400,
proto = {
dccp = {
role = "\377\327",
state = 116 't',
last_pkt = 113 'q',
last_dir = 255 '\377',
handshake_seq = 15620773601466063744
},
sctp = {
state = 1903482879,
vtag = {1903499519, 3087232}
},
tcp = {
seen = {{
td_end = 1903482879,
td_maxend = 1903499519,
td_maxwin = 3087232,
td_maxack = 3636994772,
td_scale = 7 '\a',
flags = 39 '\''
}, {
td_end = 3636994772,
td_maxend = 3640082004,
td_maxwin = 16640,
td_maxack = 1903482879,
td_scale = 7 '\a',
flags = 35 '#'
}},
state = 7 '\a',
last_dir = 0 '\000',
retrans = 0 '\000',
last_index = 3 '\003',
last_seq = 1903482879,
last_ack = 3636994772,
last_end = 1903482879,
last_win = 7040,
last_wscale = 0 '\000',
last_flags = 0 '\000'
},
gre = {
stream_timeout = 1903482879,
timeout = 1903499519
}
}
}
crash>
We've also been seeing this panic on several of our bare metal machines. Unfortunately I'm unable to gather crashdumps on these systems due to and unrelated hardware issue. Kernel version: 3.13.0-24-generic SMP i686 Distro: Ubuntu trusty We're seeing this when using a bridge to nat lxc containers with libvirt. The following sysctl tunings seem to have alleviated the symptoms for us, though we're still testing: sysctl -w net.netfilter.nf_conntrack_max=1048576 sysctl -w net.nf_conntrack_max=1048576 This has also been reproduced at Heroku with the v3.15-rc2 kernel: [17345307.967478] BUG: unable to handle kernel paging request at ffffc90003777a70 [17345307.967497] IP: [<ffffffffa013f0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [17345307.967510] PGD 1b6425067 PUD 1b6426067 PMD 1b0aed067 PTE 0 [17345307.967519] Oops: 0002 [#1] SMP [17345307.967525] Modules linked in: xt_nat veth tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat bridge stp llc xt_owner isofs ipt_REJECT xt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter ip_tables x_tables dm_crypt microcode raid10 raid456 async_memcpy async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear [17345307.967578] CPU: 0 PID: 6 Comm: kworker/u16:0 Not tainted 3.15.0-031500rc2-generic #201404201435 [17345307.967591] Workqueue: netns cleanup_net [17345307.967596] task: ffff8801b39d0000 ti: ffff8801b39cc000 task.ti: ffff8801b39cc000 [17345307.967601] RIP: e030:[<ffffffffa013f0b6>] [<ffffffffa013f0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [17345307.967611] RSP: e02b:ffff8801b39cdc48 EFLAGS: 00010246 [17345307.967617] RAX: 0000000000000000 RBX: ffff8801b1cf7b10 RCX: ffff880003110000 [17345307.967624] RDX: ffffc90003777a70 RSI: 0000000000000200 RDI: ffffffffa01434e0 [17345307.967630] RBP: ffff8801b39cdc58 R08: 0000000058690aeb R09: 00000000e834b0f3 [17345307.967636] R10: ffff880003110070 R11: 0000000000000002 R12: ffff8801b1cf7a80 [17345307.967643] R13: ffff880003110000 R14: 0000000000000000 R15: 0000000000000000 [17345307.967655] FS: 00007fd6682cc740(0000) GS:ffff8801bec00000(0000) knlGS:0000000000000000 [17345307.967662] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [17345307.967667] CR2: ffffc90003777a70 CR3: 00000000058f9000 CR4: 0000000000002660 [17345307.967673] Stack: [17345307.967677] ffffffff8176ce52 0000000000000001 ffff8801b39cdc88 ffffffffa00d1013 [17345307.967686] 0000000000000000 ffff880003110000 ffff8800ca7b8000 ffffffffa00e5140 [17345307.967694] ffff8801b39cdca8 ffffffffa00c867e ffff880003110000 ffff8800ca7b8000 [17345307.967703] Call Trace: [17345307.967712] [<ffffffff8176ce52>] ? _raw_spin_lock+0x12/0x50 [17345307.967726] [<ffffffffa00d1013>] __nf_ct_ext_destroy+0x43/0x60 [nf_conntrack] [17345307.967736] [<ffffffffa00c867e>] nf_conntrack_free+0x2e/0x70 [nf_conntrack] [17345307.967746] [<ffffffffa00c946e>] destroy_conntrack+0x9e/0xf0 [nf_conntrack] [17345307.967756] [<ffffffffa00cdc40>] ? nf_conntrack_helper_fini+0x30/0x30 [nf_conntrack] [17345307.967766] [<ffffffff81686617>] nf_conntrack_destroy+0x17/0x20 [17345307.967775] [<ffffffffa00c9358>] nf_ct_iterate_cleanup+0x78/0xb0 [nf_conntrack] [17345307.967786] [<ffffffffa00cdd1d>] nf_ct_l3proto_pernet_unregister+0x1d/0x20 [nf_conntrack] [17345307.967796] [<ffffffffa00e355d>] ipv4_net_exit+0x1d/0x60 [nf_conntrack_ipv4] [17345307.967804] [<ffffffff8164b918>] ops_exit_list.isra.1+0x38/0x60 [17345307.967811] [<ffffffff8164c222>] cleanup_net+0x112/0x230 [17345307.967820] [<ffffffff81085e2f>] process_one_work+0x17f/0x4c0 [17345307.967827] [<ffffffff81086d7b>] worker_thread+0x11b/0x3d0 [17345307.967833] [<ffffffff81086c60>] ? manage_workers.isra.21+0x190/0x190 [17345307.967841] [<ffffffff8108de69>] kthread+0xc9/0xe0 [17345307.967846] [<ffffffff8108dda0>] ? flush_kthread_worker+0xb0/0xb0 [17345307.967854] [<ffffffff8177647c>] ret_from_fork+0x7c/0xb0 [17345307.967860] [<ffffffff8108dda0>] ? flush_kthread_worker+0xb0/0xb0 [17345307.967865] Code: b7 58 12 66 85 db 74 46 0f b7 db 48 01 c3 48 83 7b 10 00 74 39 48 c7 c7 e0 34 14 a0 e8 14 db 62 e1 48 8b 03 48 8b 53 08 48 85 c0 <48> 89 02 74 04 48 89 50 08 48 ba 00 02 20 00 00 00 ad de 48 c7 [17345307.967917] RIP [<ffffffffa013f0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [17345307.967925] RSP <ffff8801b39cdc48> [17345307.967928] CR2: ffffc90003777a70 [17345307.967933] ---[ end trace 84d4f3185a40459f ]--- [17345307.967938] Kernel panic - not syncing: Fatal exception in interrupt This also happens with the (almost) current mainline kernel, from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/ Commit for this build was: ed8c37e158cb697df905d6b4933bc107c69e8936 Traceback (most recent call last): File "/usr/bin/cloud-init", line 618, in <module> sys.exit(main()) File "/usr/bin/cloud-init", line 614, in main get_uptime=True, func=functor, args=(name, args)) File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1875, in log_time ret = func(*args, **kwargs) File "/usr/bin/cloud-init", line 510, in status_wrapper atomic_write_json(status_path, status) File "/usr/bin/cloud-init", line 434, in atomic_write_json raise e OSError: [Errno 2] No such file or directory: '/var/lib/cloud/data/tmpMBxCza' [474208.150506] BUG: unable to handle kernel paging request at ffffc90003661288 [474208.150524] IP: [<ffffffffa013a0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [474208.150536] PGD 1b6423067 PUD 1b6424067 PMD 1b255b067 PTE 0 [474208.150544] Oops: 0002 [#1] SMP [474208.150549] Modules linked in: xt_nat veth tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat bridge stp llc xt_owner ipt_REJECT xt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter ip_tables x_tables isofs dm_crypt raid10 raid456 async_memcpy async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear [474208.150602] CPU: 3 PID: 6 Comm: kworker/u16:0 Not tainted 3.15.0-999-generic #201404300254 [474208.150614] Workqueue: netns cleanup_net [474208.150619] task: ffff8801b39d0000 ti: ffff8801b39c6000 task.ti: ffff8801b39c6000 [474208.150625] RIP: e030:[<ffffffffa013a0b6>] [<ffffffffa013a0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [474208.150634] RSP: e02b:ffff8801b39c7c48 EFLAGS: 00010246 [474208.150639] RAX: 0000000000000000 RBX: ffff8801af4e5510 RCX: ffff8801b2040000 [474208.150645] RDX: ffffc90003661288 RSI: 0000000000000200 RDI: ffffffffa013d4e0 [474208.150651] RBP: ffff8801b39c7c58 R08: 00000000f72af2f7 R09: 0000000002eb94ae [474208.150657] R10: ffff8801b2040070 R11: 0000000000000002 R12: ffff8801af4e5480 [474208.150664] R13: ffff8801b2040000 R14: 0000000000000000 R15: 0000000000000000 [474208.150674] FS: 00007fb3072be700(0000) GS:ffff8801becc0000(0000) knlGS:0000000000000000 [474208.150681] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [474208.150686] CR2: ffffc90003661288 CR3: 00000000f54bb000 CR4: 0000000000002660 [474208.150692] Stack: [474208.150695] ffffffff81763762 0000000000000001 ffff8801b39c7c88 ffffffffa00f2f53 [474208.150704] 0000000000000000 ffff8801b2040000 ffff88008c4e0000 ffffffffa00d1140 [474208.150712] ffff8801b39c7ca8 ffffffffa00ea67e ffff8801b2040000 ffff88008c4e0000 [474208.150721] Call Trace: [474208.150729] [<ffffffff81763762>] ? _raw_spin_lock+0x12/0x50 [474208.150744] [<ffffffffa00f2f53>] __nf_ct_ext_destroy+0x43/0x60 [nf_conntrack] [474208.150755] [<ffffffffa00ea67e>] nf_conntrack_free+0x2e/0x70 [nf_conntrack] [474208.150765] [<ffffffffa00eb46e>] destroy_conntrack+0x9e/0xf0 [nf_conntrack] [474208.150775] [<ffffffffa00efbd0>] ? nf_conntrack_helper_fini+0x30/0x30 [nf_conntrack] [474208.150785] [<ffffffff8167f367>] nf_conntrack_destroy+0x17/0x20 [474208.150794] [<ffffffffa00eb358>] nf_ct_iterate_cleanup+0x78/0xb0 [nf_conntrack] [474208.150804] [<ffffffffa00efcad>] nf_ct_l3proto_pernet_unregister+0x1d/0x20 [nf_conntrack] [474208.150814] [<ffffffffa00cf53d>] ipv4_net_exit+0x1d/0x60 [nf_conntrack_ipv4] [474208.150822] [<ffffffff816450b8>] ops_exit_list.isra.1+0x38/0x60 [474208.150828] [<ffffffff816459c2>] cleanup_net+0x112/0x230 [474208.150837] [<ffffffff81087b4f>] process_one_work+0x17f/0x4c0 [474208.150843] [<ffffffff81088a7b>] worker_thread+0x11b/0x3d0 [474208.150850] [<ffffffff81088960>] ? manage_workers.isra.21+0x190/0x190 [474208.150857] [<ffffffff8108fb49>] kthread+0xc9/0xe0 [474208.150863] [<ffffffff8108fa80>] ? flush_kthread_worker+0xb0/0xb0 [474208.150871] [<ffffffff8176cc7c>] ret_from_fork+0x7c/0xb0 [474208.150877] [<ffffffff8108fa80>] ? flush_kthread_worker+0xb0/0xb0 [474208.150882] Code: b7 58 12 66 85 db 74 46 0f b7 db 48 01 c3 48 83 7b 10 00 74 39 48 c7 c7 e0 d4 13 a0 e8 24 94 62 e1 48 8b 03 48 8b 53 08 48 85 c0 <48> 89 02 74 04 48 89 50 08 48 ba 00 02 20 00 00 00 ad de 48 c7 [474208.150936] RIP [<ffffffffa013a0b6>] nf_nat_cleanup_conntrack+0x46/0x70 [nf_nat] [474208.150945] RSP <ffff8801b39c7c48> [474208.150949] CR2: ffffc90003661288 [474208.150957] ---[ end trace 7ab56606cd4d25a0 ]--- [474208.150961] Kernel panic - not syncing: Fatal exception in interrupt I came here from a downstream bug report (https://github.com/dotcloud/docker/issues/2960), I would like to help at finding a solution for this since it's biting us hard here. I will try the mitigation proposed by scottm for the time being and subscribe here to try eventual upcoming patches. Doesn't happen on 3.10.34. Created attachment 138271 [details]
Don't touch nat_bysource hash chain when netns is going away, preventing crash.
I finally had some time to dig into this and I believe I've pinpointed the problem. Basically what's happening is, during netns cleanup, nf_nat_net_exit gets called before ipv4_net_exit. As I understand it, nf_nat_net_exit is supposed to kill any conntrack entries which have NAT context (through nf_ct_iterate_cleanup), but for some reason this doesn't happen (perhaps something else is still holding refs to those entries?). When ipv4_net_exit is called, conntrack entries (including those with NAT context) are cleaned up, but the nat_bysource hashtable is long gone - freed in nf_nat_net_exit. The bug happens when attempting to free a conntrack entry whose NAT hash 'prev' field points to a slot in the freed hash table (head for that bin). I'm not familiar enough with the netfilter internals to propose "the right fix", but I've attached a patch which basically skips hlist_del_rcu for the nat_bysource hash links when the whole netns is going away anyway. After applying this patch we haven't had any more crashes. I can confirm the patch prevent the crash on a reliable reproducer on ubuntu trusty kernel 3.13.0-24-generic. (In reply to Rodrigo Sampaio Vaz from comment #14) > I can confirm the patch prevent the crash on a reliable reproducer on ubuntu > trusty kernel 3.13.0-24-generic. Would you mind testing http://patchwork.ozlabs.org/patch/357147/raw/ as well? This should avoid panic as well without altering the nfct destroy callback. @Florian I have built .deb packages for both patches (yours and previous one: https://github.com/gdm85/tenku/releases) and I am now testing yours; I do not have a clear cut testcase so I will just intensify the bridge activity that has triggered this behaviour on the development server (creating bridged virtual ethernet devices and then quickly destroying them). (In reply to Florian Westphal from comment #15) > (In reply to Rodrigo Sampaio Vaz from comment #14) > > I can confirm the patch prevent the crash on a reliable reproducer on > ubuntu > > trusty kernel 3.13.0-24-generic. > > Would you mind testing > > http://patchwork.ozlabs.org/patch/357147/raw/ > > as well? This should avoid panic as well without altering > the nfct destroy callback. Yes confirmed, this patch also prevent crashes with the same test case. (In reply to Florian Westphal from comment #15) > (In reply to Rodrigo Sampaio Vaz from comment #14) > > I can confirm the patch prevent the crash on a reliable reproducer on > ubuntu > > trusty kernel 3.13.0-24-generic. > > Would you mind testing > > http://patchwork.ozlabs.org/patch/357147/raw/ > > as well? This should avoid panic as well without altering > the nfct destroy callback. Good for me too. |