Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Fri, 26 Dec 2008 22:17:29 -0800 (PST) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12306
> 
>            Summary: unable to handle kernel paging request at
>                     ffff8900c6685e00
>            Product: Drivers
>            Version: 2.5
>      KernelVersion: Linux host.name.hiden 2.6.28 #1 SMP Thu Dec 25 07:28:00
>                     MSK 2008
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: drivers_other@kernel-bugs.osdl.org
>         ReportedBy: antonprk@mail.ru

2.6.28 crashed in tcp_recvmsg.

> 
> Latest working kernel version: none
> Earliest failing kernel version: 2.6.26.5

And apparently 2.6.26.5 does the same thing?

I would be suspecting a hardware failure here.  Can we have more
details please?  Some additional oops traces from this or earlier
kenrels might help.

Is the failure seen on other machines, or just this one?


> Distribution:
> Hardware Environment: Intel(R) Core(TM)2 CPU 6320  @ 1.86GHz
> Software Environment: Web Server
> Problem Description: Server crash
> 
> Kernel report in syslog:
> kernel: dcpumon[26379]: segfault at 7fff225fd1c8 ip 00000000004c142f sp
> 00007fff225e01f0 error 6 in dcpumon[400000+d3000]
> kernel: 3w-9xxx: scsi0: ERROR: (0x03:0x010D): Invalid field in CDB:.
> kernel: mkbb.sh[5721]: segfault at 10000000000 ip 0000000000425080 sp
> 00007fffb30e2560 error 4 in bash[400000+ab000]
> kernel: BUG: unable to handle kernel paging request at ffff8900c6685e00
> kernel: IP: [<ffffffff804f57e5>] tcp_recvmsg+0x265/0xca0
> kernel: PGD 0 
> kernel: Oops: 0002 [#1] SMP 
> kernel: last sysfs file:
>
> /sys/devices/pci0000:00/0000:00:1c.0/0000:09:00.0/0000:0a:01.0/host0/target0:0:0/0:0:0:0/vendor
> kernel: CPU 0 
> kernel: Modules linked in: piix ide_core
> kernel: Pid: 31968, comm: httpd Not tainted 2.6.28 #1
> kernel: RIP: 0010:[<ffffffff804f57e5>]  [<ffffffff804f57e5>]
> tcp_recvmsg+0x265/0xca0
> kernel: RSP: 0018:ffff8800d1a63c98  EFLAGS: 00010296
> kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000c0000100
> kernel: RDX: 0000000000000000 RSI: ffff8800d1a63c38 RDI: ffff8800ca15caf0
> kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: 0000000000000000
> kernel: R10: ffff880219dd5da8 R11: 0000000000000000 R12: ffff8900c66859c0
> kernel: R13: ffff8800c6685a68 R14: ffff8800c6685dac R15: 00000000ffffff95
> kernel: FS:  00007f7c10d286e0(0000) GS:ffffffff80757500(0000)
> knlGS:0000000000000000
> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> kernel: CR2: ffff8900c6685e00 CR3: 0000000019bb3000 CR4: 00000000000006e0
> kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> kernel: Process httpd (pid: 31968, threadinfo ffff8800d1a62000, task
> ffff880148778050)
> kernel: Stack:
> kernel:  00000001000000be 0000000000000001 ffff880148778050 ffff8800d1a63e68
> kernel:  0000000000000000 0000000000001000 ffff8800d1a63d80 00000000000000be
> kernel:  0000000000bb4b84 7fffffffffffffff 000005b400002238 ffff8800d1a63d80
> kernel: Call Trace:
> kernel:  [<ffffffff804b40b0>] sock_common_recvmsg+0x30/0x50
> kernel:  [<ffffffff804af9df>] sock_aio_read+0x10f/0x120
> kernel:  [<ffffffff8029e73b>] do_sync_read+0xdb/0x140
> kernel:  [<ffffffff802521e0>] autoremove_wake_function+0x0/0x30
> kernel:  [<ffffffff8034468e>] _atomic_dec_and_lock+0x4e/0x80
> kernel:  [<ffffffff802b4981>] mntput_no_expire+0x21/0x120
> kernel:  [<ffffffff8024aa8f>] do_sigaction+0xff/0x1c0
> kernel:  [<ffffffff8029e901>] vfs_read+0x161/0x170
> kernel:  [<ffffffff8029ec13>] sys_read+0x53/0x90
> kernel:  [<ffffffff8020b61b>] system_call_fastpath+0x16/0x1b
> kernel: Code: 24 20 04 00 00 0f 84 c6 06 00 00 8b 54 24 08 39 54 24 20 0f 8c
> 80
> 03 00 00 4c 89 e7 e8 85 d9 fb ff 31 f6 4c 89 e7 e8 9b d8 fb ff <41> c7 84 24
> 40
> 04 00 00 00 00 00 00 48 83 7c 24 10 00 74 4f 8b 
> kernel: RIP  [<ffffffff804f57e5>] tcp_recvmsg+0x265/0xca0
> kernel:  RSP <ffff8800d1a63c98>
> kernel: CR2: ffff8900c6685e00
> kernel: ---[ end trace 7b4209a86045831c ]---
> 
> 
> 
> Other:
> #cat
>
> /sys/devices/pci0000:00/0000:00:1c.0/0000:09:00.0/0000:0a:01.0/host0/target0:0:0/0:0:0:0/vendor
> 
> AMCC    
> 
> 
> # free -m
>              total       used       free     shared    buffers     cached
> Mem:          8005       7960         44          0        560       6622
> -/+ buffers/cache:        777       7227
> Swap:        16386         34      16352
> 
> #cat /etc/redhat-release
> 
> CentOS release 4.7 (Final)
> 

> 2.6.28 crashed in tcp_recvmsg.
> > 
> > Latest working kernel version: none
> > Earliest failing kernel version: 2.6.26.5
> And apparently 2.6.26.5 does the same thing?

Yes.

> I would be suspecting a hardware failure here.  Can we have more details
> please?  
 lspci
00:00.0 Host bridge: Intel Corporation E7230/3000/3010 Memory Controller Hub (rev c0)
00:01.0 PCI bridge: Intel Corporation E7230/3000/3010 PCI Express Root Port (rev c0)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
09:00.0 PCI bridge: Intel Corporation 6702PXH PCI Express-to-PCI Bridge A (rev 09)
09:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A (rev 09)
0a:01.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (rev 03)
0e:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
0f:00.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)



> Some additional oops traces from this or earlier kenrels might help.
BUG: unable to handle kernel paging request at ffff8900be3a6600
IP: [<ffffffff804777b9>] tcp_recvmsg+0x420/0xaee
PGD 0 
Oops: 0002 [1] SMP 
CPU 0 
Modules linked in: piix ide_disk ide_core
Pid: 22164, comm: exim Not tainted 2.6.27.8 #1
RIP: 0010:[<ffffffff804777b9>]  [<ffffffff804777b9>] tcp_recvmsg+0x420/0xaee
RSP: 0000:ffff8800d3263cb8  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff8900be3a61c0 RCX: 00000000c0000100
RDX: ffff8800d3263c70 RSI: ffff8800d3263c58 RDI: ffff880101da85f0
RBP: 00000000ffffff95 R08: ffff8800d3262000 R09: ffff8801d69cc0d8
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000002000
FS:  00007f65898bc6e0(0000) GS:ffffffff8069b5c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff8900be3a6600 CR3: 0000000079b47000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process exim (pid: 22164, threadinfo ffff8800d3262000, task ffff8801ea516c10)
Stack:  0000000000000001 ffff8801ea516c10 ffff8800be3a65ac 0000000000000000
 ffff8800d3263d90 0000000000000000 00007f65898d4137 7fffffffffffffff
 0000055000007f80 ffff8800d3263d90 ffff8800d3263df8 ffff8800d3263ee8
Call Trace:
 [<ffffffff804415d7>] ? sock_common_recvmsg+0x30/0x43
 [<ffffffff8043ddf9>] ? sock_aio_read+0xe2/0xf1
 [<ffffffff802ca7a6>] ? ext3_file_write+0x16/0x98
 [<ffffffff80281c7b>] ? do_sync_read+0xc8/0x10b
 [<ffffffff80248e18>] ? getnstimeofday+0x36/0x94
 [<ffffffff80244715>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80236b2f>] ? do_setitimer+0x176/0x32d
 [<ffffffff80281d7d>] ? vfs_read+0xbf/0x12f
 [<ffffffff8028206e>] ? sys_read+0x45/0x6e
 [<ffffffff8020b31b>] ? system_call_fastpath+0x16/0x1b




Code: 24 18 7c 14 48 89 df e8 b0 91 fc ff 31 f6 48 89 df e8 ef 90 fc ff eb 0d 48 8d 74 24 38 48 89 df e8 28 92 fc ff 48 83 7c 24 08 00 <c7> 83 40 04 00 00 00 00 00 00 0f 84 8e 00 00 00 44 89 f9 2b 8b 


----------------------------------------

BUG: unable to handle kernel paging request at ffff89021d9f7798
IP: [<ffffffff8022b089>] try_to_wake_up+0x3a/0x18e
PGD 0 
Oops: 0000 [2] SMP 
CPU 1 
Modules linked in: piix ide_disk ide_core
Pid: 7015, comm: mysqld Tainted: G      D   2.6.27.4 #1
RIP: 0010:[<ffffffff8022b089>]  [<ffffffff8022b089>] try_to_wake_up+0x3a/0x18e
RSP: 0018:ffff88006cf3bbe8  EFLAGS: 00010002
RAX: 0000000000001eff RBX: ffff89021d9f7790 RCX: 0000000000000000
RDX: ffff880028033a00 RSI: 0000000000000001 RDI: ffff89021d9f7790
RBP: ffff88006cf3bc28 R08: ffff88006cecfcc8 R09: ffff88021d40abc0
R10: 00007f4318d261eb R11: ffff88021d40abc0 R12: ffff88021bfe4d30
R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
FS:  0000000045324960(0063) GS:ffff88021f08fbc0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff89021d9f7798 CR3: 000000021b4c9000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mysqld (pid: 7015, threadinfo ffff88006cf3a000, task ffff88021f170e50)
Stack:  ffff88021d40abc0 000000001d40abc0 00000000ffffffff ffff88006cecfcc8
 ffff88021bfe4d30 0000000000000000 ffff88021bfe4d20 0000000000000000
 ffff88006cf3bc78 ffffffff8024470e 0000000000000001 ffffffff8022cb14
Call Trace:
 [<ffffffff8024470e>] autoremove_wake_function+0x9/0x2e
 [<ffffffff8022cb14>] ? __wake_up_common+0x4a/0x75
 [<ffffffff8022cbea>] __wake_up_sync+0x4a/0x6d
 [<ffffffff8044003a>] sock_def_readable+0x32/0x5d
 [<ffffffff804ad7a3>] ? unix_stream_sendmsg+0x208/0x2be
 [<ffffffff8043d2ce>] ? sock_aio_write+0xd6/0xe5
 [<ffffffff80281a8d>] ? do_sync_write+0xc8/0x10b
 [<ffffffff80289359>] ? path_walk+0x81/0x8e
 [<ffffffff80244705>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80281b91>] ? vfs_write+0xc1/0x131
 [<ffffffff80281cb4>] ? sys_write+0x45/0x6e
 [<ffffffff8020b31b>] ? system_call_fastpath+0x16/0x1b


Code: f5 41 54 53 48 89 fb 48 83 ec 18 8b 05 fd 42 47 00 c7 45 cc 00 00 00 00 a8 40 44 0f 44 75 cc f6 c4 08 74 4a 48 8b 15 ff f6 46 00 <48> 8b 47 08 65 8b 0c 25 24 00 00 00 48 63 c9 48 8b 14 ca 8b 70 



> Is the failure seen on other machines, or just this one?
On one server
Similar configurations are not present more

Andrew Morton <akpm@linux-foundation.org> wrote:
>
>> kernel: BUG: unable to handle kernel paging request at ffff8900c6685e00
>> kernel: IP: [<ffffffff804f57e5>] tcp_recvmsg+0x265/0xca0
>> kernel: PGD 0 
>> kernel: Oops: 0002 [#1] SMP 
>> kernel: last sysfs file:
>>
>> /sys/devices/pci0000:00/0000:00:1c.0/0000:09:00.0/0000:0a:01.0/host0/target0:0:0/0:0:0:0/vendor
>> kernel: CPU 0 
>> kernel: Modules linked in: piix ide_core
>> kernel: Pid: 31968, comm: httpd Not tainted 2.6.28 #1
>> kernel: RIP: 0010:[<ffffffff804f57e5>]  [<ffffffff804f57e5>]
>> tcp_recvmsg+0x265/0xca0

Does it help if you disable CONFIG_NET_DMA?

Thanks,

if disable CONFIG_NET_DMA oops is not.

New Oops
CONFIG_NET_DMA is disable 

The Oops is below:
BUG: unable to handle kernel paging request at ffff89021ddc83c4
IP: [<ffffffff8024f04c>] put_pid+0xc/0x40
PGD 0 
Oops: 0000 [#2] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:09:00.0/0000:0a:01.0/host0/target0:0:0/0:0:0:0/vendor
CPU 0 
Modules linked in: piix ide_core
Pid: 4652, comm: httpd Tainted: G      D    2.6.28-atlex #3
RIP: 0010:[<ffffffff8024f04c>]  [<ffffffff8024f04c>] put_pid+0xc/0x40
RSP: 0018:ffff880219003c28  EFLAGS: 00010286
RAX: ffff88021f04e800 RBX: ffff8800d16ebcb0 RCX: ffff8800d16eb820
RDX: ffff89021ddc83c0 RSI: 0000000000000000 RDI: ffff89021ddc83c0
RBP: ffff8800d16ebcb0 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff802b1580 R12: 0000000000000000
R13: ffff88021f04e800 R14: ffff88021f04e910 R15: ffff880219003ca8
FS:  00007f79083ae6e0(0000) GS:ffffffff80756500(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff89021ddc83c4 CR3: 0000000219cea000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process httpd (pid: 4652, threadinfo ffff880219002000, task ffff88021f2488b0)
Stack:
 ffff880219003ca8 ffffffff802e5d10 ffff88005e93ea08 ffff8800d16ebcb0
 ffffffff802e5cf0 ffffffff802b162b ffff88003f7428e0 ffff88003f7428a0
 ffff88003f7428e0 ffffffff802ae622 ffff8801b6778be0 ffff8801b6778be0
Call Trace:
 [<ffffffff802e5d10>] ? proc_delete_inode+0x20/0x70
 [<ffffffff802e5cf0>] ? proc_delete_inode+0x0/0x70
 [<ffffffff802b162b>] ? generic_delete_inode+0xab/0x150
 [<ffffffff802ae622>] ? d_kill+0x42/0x70
 [<ffffffff802aebc9>] ? __shrink_dcache_sb+0x279/0x310
 [<ffffffff802af086>] ? shrink_dcache_parent+0xf6/0x110
 [<ffffffff802e9aaf>] ? proc_flush_task+0x24f/0x290
 [<ffffffff802ad6fe>] ? core_sys_select+0x29e/0x370
 [<ffffffff8023e9aa>] ? release_task+0x1a/0x310
 [<ffffffff8024079e>] ? wait_consider_task+0x6ae/0x8e0
 [<ffffffff80212519>] ? read_tsc+0x9/0x20
 [<ffffffff80240b52>] ? do_wait+0x182/0x3b0
 [<ffffffff80235750>] ? default_wake_function+0x0/0x10
 [<ffffffff80235750>] ? default_wake_function+0x0/0x10
 [<ffffffff802ad8ae>] ? sys_select+0xde/0x110
 [<ffffffff80240ed6>] ? sys_wait4+0x86/0xd0
 [<ffffffff8020b61b>] ? system_call_fastpath+0x16/0x1b
Code: 41 5c b8 ff ff ff ff c3 5b 8d 84 28 00 c0 ff ff 5d 41 5c c3 66 66 66 90 66 66 66 90 66 66 90 48 83 ec 08 48 85 ff 48 89 fa 74 2a <8b> 47 04 48 c1 e0 05 83 3f 01 48 8b 4c 38 38 74 0a f0 ff 0f 0f