Bug 20682

Summary:	Kernel bug, possible double free, effecting kernel.org machines
Product:	Memory Management	Reporter:	John 'Warthog9' Hawley (warthog9)
Component:	Slab Allocator	Assignee:	Andrew Morton (akpm)
Status:	RESOLVED DUPLICATE
Severity:	normal
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:		Subsystem:
Regression:	No	Bisected commit-id:

Description John 'Warthog9' Hawley 2010-10-18 19:02:22 UTC

Discovered this on one of the kernel.org machines - it's been happening semi-consistently on a pair of boxes.  Seems like a double free somewhere and at that point the whole box falls over dead basically.

------------[ cut here ]------------
kernel BUG at mm/slub.c:2835!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/kernel/mm/ksm/run
CPU 1 
Modules linked in: ocfs2 mptctl mptbase drbd lru_cache nfsd lockd nfs_acl auth_rpcgss sunrpc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs cpufreq_ondemand powernow_k8 freq_table 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 xfs exportfs tg3 hpwdt amd64_edac_mod i2c_amd756 i2c_core edac_core shpchp k8temp amd_rng edac_mce_amd microcode pata_acpi ata_generic cciss pata_amd [last unloaded: scsi_wait_scan]

Pid: 1713, comm: snmpd Not tainted 2.6.34.7-56.fc13.x86_64 #1 /ProLiant DL385 G1
RIP: 0010:[<ffffffff811006d6>]  [<ffffffff811006d6>] kfree+0x5e/0xcb
RSP: 0018:ffff8801f6433df8  EFLAGS: 00010246
RAX: 0040000000000400 RBX: ffff8803ed0eb9b0 RCX: ffff8803e9c92340
RDX: ffffea0000000000 RSI: ffffea0003800000 RDI: ffff880100000002
RBP: ffff8801f6433e18 R08: ffff8803e9c92958 R09: 0000000000000000
R10: 0000000000000011 R11: 0000000000000246 R12: ffff880100000002
R13: ffffffff81125e27 R14: ffffffff8115dbdc R15: ffff8803dd19ea80
FS:  00007ff4a31917a0(0000) GS:ffff880207400000(0000) knlGS:00000000f76fa6d0
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff4a31b2000 CR3: 00000001f6455000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process snmpd (pid: 1713, threadinfo ffff8801f6432000, task ffff8801f5688000)
Stack:
 ffff8801f6433e18 ffff8803ed0eb9b0 ffff8803f61ec480 ffff8803ed0eb9b0
<0> ffff8801f6433e48 ffffffff81125e27 ffff8801f6433e38 ffff8803dd19ea80
<0> ffff8803ed0eb9b0 ffff8803e9c92940 ffff8801f6433e78 ffffffff8115dc10
Call Trace:
 [<ffffffff81125e27>] seq_release_private+0x28/0x44
 [<ffffffff8115dc10>] seq_release_net+0x34/0x3d
 [<ffffffff81155ada>] proc_reg_release+0xd3/0xf0
 [<ffffffff8110efbb>] __fput+0x12a/0x1dc
 [<ffffffff8110f087>] fput+0x1a/0x1c
 [<ffffffff8110c0f7>] filp_close+0x68/0x72
 [<ffffffff8110c19e>] sys_close+0x9d/0xd2
 [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b
Code: ef ff 13 48 83 c3 08 48 83 3b 00 eb ec 49 83 fc 10 76 7d 4c 89 e7 e8 67 e4 ff ff 48 89 c6 48 8b 00 84 c0 78 14 66 a9 00 c0 75 04 <0f> 0b eb fe 48 89 f7 e8 66 36 fd ff eb 57 48 8b 4d 08 48 8b 7e 
RIP  [<ffffffff811006d6>] kfree+0x5e/0xcb
 RSP <ffff8801f6433df8>
---[ end trace 1a4b1fd758dd1fdb ]---
general protection fault: 0000 [#2] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:19.3/name
CPU 1 
Modules linked in: ocfs2 mptctl mptbase drbd lru_cache nfsd lockd nfs_acl auth_rpcgss sunrpc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs cpufreq_ondemand powernow_k8 freq_table 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 xfs exportfs tg3 hpwdt amd64_edac_mod i2c_amd756 i2c_core edac_core shpchp k8temp amd_rng edac_mce_amd microcode pata_acpi ata_generic cciss pata_amd [last unloaded: scsi_wait_scan]

Pid: 16274, comm: snmpd Tainted: G      D    2.6.34.7-56.fc13.x86_64 #1 /ProLiant DL385 G1
RIP: 0010:[<ffffffff8110136b>]  [<ffffffff8110136b>] __kmalloc_track_caller+0xe3/0x14c
RSP: 0018:ffff8800380dbcc8  EFLAGS: 00010006
RAX: 0000000000000000 RBX: 0003000000000000 RCX: 000000000000000b
RDX: 0000000100000000 RSI: 00000000000006a9 RDI: ffffffff8177d901
RBP: ffff8800380dbd18 R08: ffff880207412570 R09: ffff8800380dbe88
R10: ffff8800380dbf28 R11: 0000000000000000 R12: ffffffff81a28520
R13: 000000000000000b R14: 00000000000000d0 R15: 00000000000000d0
FS:  00007f4afe3857a0(0000) GS:ffff880207400000(0000) knlGS:00000000f6d92b70
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4afc38cd50 CR3: 00000000f1fe8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process snmpd (pid: 16274, threadinfo ffff8800380da000, task ffff8801ccdcddc0)
Stack:
 ffff8800380dbcf8 0000000000000003 ffffffff81117373 0000000000000246
<0> ffff8800380dbd18 ffff8803ed0178e0 ffff8803ed017840 ffff8801f0938a50
<0> ffff8801f09046c0 0000000000000000 ffff8800380dbd48 ffffffff810dd233
Call Trace:
 [<ffffffff81117373>] ? vfs_rename+0xb2/0x3e1
 [<ffffffff810dd233>] kstrdup+0x31/0x49
 [<ffffffff81117373>] vfs_rename+0xb2/0x3e1
 [<ffffffff81116523>] ? __lookup_hash+0x55/0xf1
 [<ffffffff811ce973>] ? security_inode_permission+0x21/0x23
 [<ffffffff81118cfd>] sys_renameat+0x193/0x20c
 [<ffffffff810e4e5a>] ? handle_mm_fault+0x452/0x97b
 [<ffffffff81450195>] ? do_page_fault+0x28e/0x2bb
 [<ffffffff81118d91>] sys_rename+0x1b/0x1d
 [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b
Code: 90 66 90 48 89 45 c8 fa 66 66 90 66 66 90 65 4c 8b 04 25 90 e8 00 00 49 8b 04 24 49 01 c0 49 8b 18 48 85 db 74 0e 49 63 44 24 18 <48> 8b 04 03 49 89 00 eb 15 48 8b 4d c0 83 ca ff 44 89 fe 4c 89 
RIP  [<ffffffff8110136b>] __kmalloc_track_caller+0xe3/0x14c
 RSP <ffff8800380dbcc8>
---[ end trace 1a4b1fd758dd1fdc ]---
block drbd1: write: error=-95 s=1658s
block drbd1: Method to ensure write ordering: flush
block drbd1: local disk flush failed with status -95
block drbd1: Method to ensure write ordering: drain
o2net: accepted connection from node demeter2.kernel.org (num 2) at 172.20.0.20:7777
ocfs2_dlm: Node 2 joins domain FC86A681BA714C7AA126836FFC1D4C8C
ocfs2_dlm: Nodes in domain ("FC86A681BA714C7AA126836FFC1D4C8C"): 1 2 
general protection fault: 0000 [#3] SMP 
last sysfs file: /sys/kernel/mm/ksm/run
CPU 1 
Modules linked in: ocfs2 mptctl mptbase drbd lru_cache nfsd lockd nfs_acl auth_rpcgss sunrpc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs cpufreq_ondemand powernow_k8 freq_table 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 xfs exportfs tg3 hpwdt amd64_edac_mod i2c_amd756 i2c_core edac_core shpchp k8temp amd_rng edac_mce_amd microcode pata_acpi ata_generic cciss pata_amd [last unloaded: scsi_wait_scan]

Pid: 29724, comm: lsof Tainted: G      D    2.6.34.7-56.fc13.x86_64 #1 /ProLiant DL385 G1
RIP: 0010:[<ffffffff81101615>]  [<ffffffff81101615>] __kmalloc+0xeb/0x150
RSP: 0018:ffff8801c9cd5be8  EFLAGS: 00010006
RAX: 0000000000000000 RBX: 0003000000000000 RCX: ffffffff811264ff
RDX: 0000000100000000 RSI: 00000000000006a9 RDI: ffffffff8177d901
RBP: ffff8801c9cd5c28 R08: ffff880207412570 R09: ffff8803ea9aed80
R10: ffff8803ce0767c0 R11: 0000000000000000 R12: ffffffff81a28520
R13: 0000000000000010 R14: 00000000000080d0 R15: 00000000000080d0
FS:  00007f364411c7a0(0000) GS:ffff880207400000(0000) knlGS:00000000f5990b70
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004091c0 CR3: 000000019f488000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process lsof (pid: 29724, threadinfo ffff8801c9cd4000, task ffff8801cd6c8000)
Stack:
 ffffffff811264ff 0000000000000246 ffffc900013ed638 ffff8803ee7dfc00
<0> ffffffff81666660 ffff8803ce0765b0 ffff8801f6bed1f8 ffff8803ee7dfc00
<0> ffff8801c9cd5c58 ffffffff811264ff ffff8801c9cd5c58 ffffffff81e074e0
Call Trace:
 [<ffffffff811264ff>] ? __seq_open_private+0x25/0x5f
 [<ffffffff811264ff>] __seq_open_private+0x25/0x5f
 [<ffffffff8115dcf9>] seq_open_net+0x65/0x8c
 [<ffffffff814217b6>] unix_seq_open+0x1a/0x1c
 [<ffffffff81155f64>] proc_reg_open+0xd7/0x163
 [<ffffffff8115dbdc>] ? seq_release_net+0x0/0x3d
 [<ffffffff81155e8d>] ? proc_reg_open+0x0/0x163
 [<ffffffff8110c4a1>] __dentry_open+0x173/0x2aa
 [<ffffffff811ce973>] ? security_inode_permission+0x21/0x23
 [<ffffffff8110c6a7>] nameidata_to_filp+0x3f/0x50
 [<ffffffff81117e21>] do_last+0x447/0x5b8
 [<ffffffff81119868>] do_filp_open+0x217/0x5fe
 [<ffffffff81214e23>] ? might_fault+0x21/0x23
 [<ffffffff811225ca>] ? alloc_fd+0x7b/0x124
 [<ffffffff8110c236>] do_sys_open+0x63/0x10f
 [<ffffffff8110c315>] sys_open+0x20/0x22
 [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b
Code: 90 66 90 48 89 45 c8 fa 66 66 90 66 66 90 65 4c 8b 04 25 90 e8 00 00 49 8b 04 24 49 01 c0 49 8b 18 48 85 db 74 0e 49 63 44 24 18 <48> 8b 04 03 49 89 00 eb 11 83 ca ff 44 89 fe 4c 89 e7 e8 16 eb 
RIP  [<ffffffff81101615>] __kmalloc+0xeb/0x150
 RSP <ffff8801c9cd5be8>
---[ end trace 1a4b1fd758dd1fdd ]---

Comment 1 John 'Warthog9' Hawley 2010-11-24 22:53:35 UTC


*** This bug has been marked as a duplicate of bug 20702 ***