Bug 11543
| Summary: | kernel panic: softlockup in tick_periodic() ??? | ||
|---|---|---|---|
| Product: | Platform Specific/Hardware | Reporter: | Joshua Hoblitt (j_kernel) |
| Component: | x86-64 | Assignee: | platform_x86_64 (platform_x86_64) |
| Status: | CLOSED INSUFFICIENT_DATA | ||
| Severity: | normal | CC: | gorcunov, rjw |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 2.6.27-rc4-21704-gd25e26b | Subsystem: | |
| Regression: | Yes | Bisected commit-id: | |
| Bug Depends on: | |||
| Bug Blocks: | 11167 | ||
| Attachments: |
dmesg
acpi table dump kernel panics from 2008-10-17 -> 2008-11-13 2.6.27-rc8 ,config |
||
Created attachment 17734 [details]
dmesg
I just noticed that there are warnings in the dmesg about the NMI watchdog. Is this the watchdog malfunctioning? [ 0.456700] Testing NMI watchdog ... [ 0.529091] WARNING: CPU#0: NMI appears to be stuck (0->0)! [ 0.529968] Please report this to bugzilla.kernel.org, [ 0.533301] and attach the output of the 'dmesg' command. [ 0.536635] [ 0.539968] WARNING: CPU#1: NMI appears to be stuck (0->0)! [ 0.543301] Please report this to bugzilla.kernel.org, [ 0.546635] and attach the output of the 'dmesg' command. [ 0.549968] [ 0.551646] WARNING: CPU#2: NMI appears to be stuck (0->0)! [ 0.553301] Please report this to bugzilla.kernel.org, [ 0.556635] and attach the output of the 'dmesg' command. [ 0.559968] [ 0.563302] WARNING: CPU#3: NMI appears to be stuck (0->0)! [ 0.566635] Please report this to bugzilla.kernel.org, [ 0.569968] and attach the output of the 'dmesg' command. Reply-To: akpm@linux-foundation.org (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Thu, 11 Sep 2008 16:46:29 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=11543 > > Summary: kernel panic: softlockup in tick_periodic() ??? > Product: Platform Specific/Hardware > Version: 2.5 > KernelVersion: 2.6.27-rc4-21704-gd25e26b > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: x86-64 > AssignedTo: platform_x86_64@kernel-bugs.osdl.org > ReportedBy: j_kernel@hoblitt.com > Is this a regression? Was 2.6.26 OK, for example? > [11532.103605] do_IRQ: 0.175 No irq handler for vector > <Sep/11 12:13 pm>[11532.103613] do_IRQ: 2.175 No irq handler for vector > <Sep/11 12:13 pm>[11532.103617] do_IRQ: 1.175 No irq handler for vector > <Sep/11 12:14 pm>[11560.779989] do_IRQ: 0.179 No irq handler for vector > <Sep/11 12:15 pm>[11622.181968] Kernel panic - not syncing: softlockup: hung > tas<Sep/11 12:15 pm> > <Sep/11 12:15 pm>[11622.181968] ------------[ cut here > ]------------ > <Sep/11 12:15 pm>[11622.181968] WARNING: at kernel/mutex.c:351 > mutex_trylock+0x45/0xf6() > <Sep/11 12:15 pm>[11622.181968] Modules linked in: w83627hf hwmon_vid autofs4 > smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs > dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx > scsi_wait_scan > <Sep/11 12:15 pm>[11622.181968] Pid: 17192, comm: ppImage Not tainted > 2.6.27-rc4-21704-gd25e26b #1 > <Sep/11 12:15 pm>[11622.181968] > <Sep/11 12:15 pm>[11622.181968] Call Trace: > <Sep/11 12:15 pm>[11622.181968] <IRQ> [<ffffffff80235319>] > warn_on_slowpath+0x51/0x77 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] > release_console_sem+0x3e/0x1a1 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff805c6031>] mutex_trylock+0x45/0xf6 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8025efc3>] crash_kexec+0x17/0xef > <Sep/11 12:15 pm>[11622.181968] [<ffffffff803bb5b9>] > bust_spinlocks+0x15/0x30 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff80235218>] panic+0x8f/0x13f > <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] > release_console_sem+0x3e/0x1a1 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] > release_console_sem+0x3e/0x1a1 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff80272145>] > softlockup_tick+0x19e/0x1ab > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8023dda4>] > update_process_times+0x26/0x4b > > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f7a4>] tick_periodic+0x6e/0x79 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f7c7>] > tick_handle_periodic+0x18/0x59 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f96a>] > tick_do_broadcast+0x4d/0x86 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024fa20>] > tick_do_periodic_broadcast+0x23/0x31 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024fa3c>] > tick_handle_periodic_broadcast+0xe/0x42 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020e9f6>] > timer_event_interrupt+0x1a/0x21 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff80272591>] > handle_IRQ_event+0x1e/0x4c > <Sep/11 12:15 pm>[11622.181968] [<ffffffff80273885>] > handle_edge_irq+0xe8/0x12b > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020e96f>] do_IRQ+0xf1/0x15e > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020c3e1>] ret_from_intr+0x0/0xa > <Sep/11 12:15 pm>[11622.181968] <EOI> [<ffffffff8021b79b>] > native_flush_tlb_others+0x64/0xb3 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b7c5>] > native_flush_tlb_others+0x8e/0xb3 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b7be>] > native_flush_tlb_others+0x87/0xb3 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b8b2>] > flush_tlb_page+0x5e/0x65 > <Sep/11 12:15 pm>[11622.181968] [<ffffffff8022531b>] > ptep_set_access_flags+0x1b/0x1f > <Sep/11 12:15 pm>[11622.181968] [<ffffffff80285193>] do_wp_page+0x48b/0x51e argh, death by wordwrapping. I can't work out who called panic(), nor why. The panic code called the kexec code which called mutex_trylock() which called spin_lock_mutex() which then stupidly went and blurted a load of debug stuff because of in_interrupt(). Something like this: --- a/include/linux/debug_locks.h~a +++ a/include/linux/debug_locks.h @@ -17,7 +17,7 @@ extern int debug_locks_off(void); ({ \ int __ret = 0; \ \ - if (unlikely(c)) { \ + if (!oops_in_progress && unlikely(c)) { \ if (debug_locks_off() && !debug_locks_silent) \ WARN_ON(1); \ __ret = 1; \ _ might prevent the debugging code from preventing us from finding bugs :( On Thu, Sep 11, 2008 at 05:02:58PM -0700, Andrew Morton wrote: > Is this a regression? Was 2.6.26 OK, for example? It might be a regression. ;) The last build we were running on this hardware was 2.6.24.2 and NMI watchdog support was not enabled. We were however experiencing random deadlocks, which I had been attributing to problems with forcedeth.c (which causes the NIC to totally crap out but not deadlock the machine) but I am now of the mind that there are multiple problems with distinct failure modes. > I can't work out who called panic(), nor why. One more data point. We booted this kernel on 14 machines this morning and only one has had this panic thus far... > The panic code called the kexec code which called mutex_trylock() which > called spin_lock_mutex() which then stupidly went and blurted a load of > debug stuff because of in_interrupt(). > > Something like this: > > --- a/include/linux/debug_locks.h~a > +++ a/include/linux/debug_locks.h > @@ -17,7 +17,7 @@ extern int debug_locks_off(void); > ({ \ > int __ret = 0; \ > \ > - if (unlikely(c)) { \ > + if (!oops_in_progress && unlikely(c)) { \ > if (debug_locks_off() && !debug_locks_silent) \ > WARN_ON(1); \ > __ret = 1; \ > _ > > might prevent the debugging code from preventing us from finding bugs :( Do you want me to give that patch a try or sit tight for a bit? -J -- Reply-To: akpm@linux-foundation.org On Thu, 11 Sep 2008 16:54:58 -1000 j_kernel@hoblitt.com wrote: > > The panic code called the kexec code which called mutex_trylock() which > > called spin_lock_mutex() which then stupidly went and blurted a load of > > debug stuff because of in_interrupt(). > > > > Something like this: > > > > --- a/include/linux/debug_locks.h~a > > +++ a/include/linux/debug_locks.h > > @@ -17,7 +17,7 @@ extern int debug_locks_off(void); > > ({ \ > > int __ret = 0; \ > > \ > > - if (unlikely(c)) { \ > > + if (!oops_in_progress && unlikely(c)) { \ > > if (debug_locks_off() && !debug_locks_silent) \ > > WARN_ON(1); \ > > __ret = 1; \ > > _ > > > > might prevent the debugging code from preventing us from finding bugs :( > > Do you want me to give that patch a try or sit tight for a bit? It's be good if you can try it please, see if we can get a cleaner trace. > It's be good if you can try it please, see if we can get a cleaner
> trace.
We also get a cleaner trace, when we switch off softlockup_panic.
But:
[ 0.003333] hpet clockevent registered
[ 0.056670] using C1E aware idle routine
Please update to 2.6.27-rc6. We fixed a couple of nasty bugs in that
area after rc4. The bugs result in hard locks or softlockup watchdog
hits.
Thanks,
tglx
* Andrew Morton <akpm@linux-foundation.org> wrote: > I can't work out who called panic(), nor why. > > The panic code called the kexec code which called mutex_trylock() > which called spin_lock_mutex() which then stupidly went and blurted a > load of debug stuff because of in_interrupt(). agreed - applied your fix in the form below to tip/master - thanks Andrew. J, you might want to try tip/master, it includes all known fixes for this area and this debug improvement as well. You can pick it up via: http://people.redhat.com/mingo/tip.git/README Ingo ----------> From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001 From: Andrew Morton <akpm@linux-foundation.org> Date: Thu, 11 Sep 2008 17:02:58 -0700 Subject: [PATCH] lock debug: sit tight when we are already in a panic in: > http://bugzilla.kernel.org/show_bug.cgi?id=11543 The panic code called the kexec code which called mutex_trylock() which called spin_lock_mutex() which then stupidly went and blurted a load of debug stuff because of in_interrupt(). Keep the lock debug code from escallating an already crappy situation. Signed-off-by: Ingo Molnar <mingo@elte.hu> --- include/linux/debug_locks.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h index 4aaa4af..096476f 100644 --- a/include/linux/debug_locks.h +++ b/include/linux/debug_locks.h @@ -17,7 +17,7 @@ extern int debug_locks_off(void); ({ \ int __ret = 0; \ \ - if (unlikely(c)) { \ + if (!oops_in_progress && unlikely(c)) { \ if (debug_locks_off() && !debug_locks_silent) \ WARN_ON(1); \ __ret = 1; \ > ------- Comment #6 from tglx@linutronix.de 2008-09-12 00:43 -------
> > It's be good if you can try it please, see if we can get a cleaner
> > trace.
>
> We also get a cleaner trace, when we switch off softlockup_panic.
>
> But:
>
> [ 0.003333] hpet clockevent registered
> [ 0.056670] using C1E aware idle routine
>
> Please update to 2.6.27-rc6. We fixed a couple of nasty bugs in that
> area after rc4. The bugs result in hard locks or softlockup watchdog
> hits.
Does -rc5 include the patches? I'm using the netdev tree, trying to fix
another problem, and it hasn't pulled -rc6 yet.
-J
--
> Does -rc5 include the patches? I'm using the netdev tree, trying to fix > another problem, and it hasn't pulled -rc6 yet. No. But there is a combopatch against rc5 at: http://bugzilla.kernel.org/attachment.cgi?id=17644 Reply-To: josh@hoblitt.com I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET patch Thomas pointed me at. I'll let it roast on these 14 machine is production over the weekend to see if we get another panic. I'm attaching the dmesg from this kernel. We're still getting the NMI watchdog warning and the rtc is [still] hosed (I think it was last working around -rc3). -J -- On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote: > > * Andrew Morton <akpm@linux-foundation.org> wrote: > > > I can't work out who called panic(), nor why. > > > > The panic code called the kexec code which called mutex_trylock() > > which called spin_lock_mutex() which then stupidly went and blurted a > > load of debug stuff because of in_interrupt(). > > agreed - applied your fix in the form below to tip/master - thanks > Andrew. > > J, you might want to try tip/master, it includes all known fixes for > this area and this debug improvement as well. You can pick it up via: > > http://people.redhat.com/mingo/tip.git/README > > Ingo > > ----------> > >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001 > From: Andrew Morton <akpm@linux-foundation.org> > Date: Thu, 11 Sep 2008 17:02:58 -0700 > Subject: [PATCH] lock debug: sit tight when we are already in a panic > > in: > > > http://bugzilla.kernel.org/show_bug.cgi?id=11543 > > The panic code called the kexec code which called mutex_trylock() which > called spin_lock_mutex() which then stupidly went and blurted a load of > debug stuff because of in_interrupt(). > > Keep the lock debug code from escallating an already crappy situation. > > Signed-off-by: Ingo Molnar <mingo@elte.hu> > --- > include/linux/debug_locks.h | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h > index 4aaa4af..096476f 100644 > --- a/include/linux/debug_locks.h > +++ b/include/linux/debug_locks.h > @@ -17,7 +17,7 @@ extern int debug_locks_off(void); > ({ \ > int __ret = 0; \ > \ > - if (unlikely(c)) { \ > + if (!oops_in_progress && unlikely(c)) { \ > if (debug_locks_off() && !debug_locks_silent) \ > WARN_ON(1); \ > __ret = 1; \ Reply-To: josh@hoblitt.com Since Friday 2 different machines have experienced crashes. One was a total deadlock with no response on the console. The other one reported the trace below on the console and stopped responding to ssh but I was able to loging via the serial console and reboot the system. This particular system has had a number of "odd" kernel traces over the last year and I'm starting to actually wonder if it may have a bad DIMM in it as occasionally the failure mode seems to be different then the deadlocks/etc. we see in the other 15 nodes with identical hardware. [30712.654542] general protection fault: 0000 [1] SMP <Sep/12 09:25 pm>[30712.657678] CPU 3 <Sep/12 09:25 pm>[30712.657678] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Sep/12 09:25 pm>[30712.657678] Pid: 1178, comm: rpciod/3 Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2 <Sep/12 09:25 pm>[30712.657678] RIP: 0010:[<ffffffff805ac57d>] [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8 <Sep/12 09:25 pm>[30712.657678] RSP: 0018:ffff88012e5d1e08 EFLAGS: 00010206 <Sep/12 09:25 pm>[30712.657678] RAX: ffffffff807adb48 RBX: ffff880126d61088 RCX: 0400000000000000 <Sep/12 09:25 pm>[30712.657678] RDX: ffff88022bcc0380 RSI: ffff88022bcc0000 RDI: ffff880126d61088 <Sep/12 09:25 pm>[30712.657678] RBP: ffff88022bc88000 R08: 0000000000000003 R09: ffff88022e038f10 <Sep/12 09:25 pm>[30712.657678] R10: 0000000000000001 R11: ffff88012e4a0048 R12: 0400000000000000 <Sep/12 09:25 pm>[30712.657678] R13: ffffffff8059f7a8 R14: ffff88022bc88610 R15: 0000000000000000 <Sep/12 09:25 pm>[30712.657678] FS: 00007f006fb306f0(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000 <Sep/12 09:25 pm>[30712.657678] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b <Sep/12 09:25 pm>[30712.657678] CR2: 00000000011e3018 CR3: 00000001f3527000 CR4:<Sep/12 09:25 pm> <Sep/12 09:25 pm>[30712.657678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <Sep/12 09:25 pm>[30712.657678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 <Sep/12 09:25 pm>[30712.657678] Process rpciod/3 (pid: 1178, threadinfo ffff88012e5d0000, task ffff88012e5acb50) <Sep/12 09:25 pm>[30712.657678] Stack: ffffffff8059b996 ffff880126d61088 00000000fffffff5 ffff880126d61118 <Sep/12 09:25 pm>[30712.657678] ffffffff8059f7a8 ffffffff808a4b20 ffffffff80599aa1 ffffffff8059f7a8 <Sep/12 09:25 pm>[30712.657678] 0000000000000000 ffff880126d61088 ffffffff8059f5f3 ffff88012e5d1e90 <Sep/12 09:25 pm>[30712.657678] Call Trace: <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059b996>] ? xprt_release+0x2c/0x172 <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc <Sep/12 09:25 pm>[30712.657678] [<ffffffff80599aa1>] ? call_reserveresult+0x95/0xd1 <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f5f3>] ? __rpc_execute+0x73/0x228 <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244394>] ? run_workqueue+0xed/0x1ed <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024433e>] ? run_workqueue+0x97/0x1ed <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244ef7>] ? worker_thread+0xd8/0xe3 <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024791a>] ? autoremove_wake_function+0x0/0x2e <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244e1f>] ? worker_thread+0x0/0xe3 <Sep/12 09:25 pm>[30712.657678] [<ffffffff80247805>] ? kthread+0x47/0x76 <Sep/12 09:25 pm>[30712.657678] [<ffffffff80230223>] ? schedule_tail+0x27/0x5f <Sep/12 09:25 pm>[30712.657678] [<ffffffff8020ce09>] ? child_rip+0xa/0x11 <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024767c>] ? kthreadd+0x167/0x18c <Sep/12 09:25 pm>[30712.657678] [<ffffffff802477be>] ? kthread+0x0/0x76 <Sep/12 09:25 pm>[30712.657678] [<ffffffff8020cdff>] ? child_rip+0x0/0x11 <Sep/12 09:25 pm>[30712.657678] <Sep/12 09:25 pm>[30712.657678] <Sep/12 09:25 pm>[30712.657678] Code: b0 98 00 00 00 48 85 f6 0f 94 c2 48 85 c9 0f 94 c0 08 c2 0f 85 94 00 00 00 48 8b 47 38 8b 50 28 48 c1 e2 07 48 8d 14 16 48 ff 02 <48> 63 81 50 01 00 00 48 01 42 08 0f b7 87 e0 00 00 00 48 01 42 <Sep/12 09:25 pm>[30712.657678] RIP [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8 <Sep/12 09:25 pm>[30712.657678] RSP <ffff88012e5d1e08> <Sep/12 09:25 pm>[30712.951578] ---[ end trace e646bef2de7f0b20 ]--- -J -- On Fri, Sep 12, 2008 at 02:13:06PM -1000, Joshua Hoblitt wrote: > I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET > patch Thomas pointed me at. I'll let it roast on these 14 machine is > production over the weekend to see if we get another panic. > > I'm attaching the dmesg from this kernel. We're still getting the NMI > watchdog warning and the rtc is [still] hosed (I think it was last > working around -rc3). > > -J > > -- > On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote: > > > > * Andrew Morton <akpm@linux-foundation.org> wrote: > > > > > I can't work out who called panic(), nor why. > > > > > > The panic code called the kexec code which called mutex_trylock() > > > which called spin_lock_mutex() which then stupidly went and blurted a > > > load of debug stuff because of in_interrupt(). > > > > agreed - applied your fix in the form below to tip/master - thanks > > Andrew. > > > > J, you might want to try tip/master, it includes all known fixes for > > this area and this debug improvement as well. You can pick it up via: > > > > http://people.redhat.com/mingo/tip.git/README > > > > Ingo > > > > ----------> > > >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001 > > From: Andrew Morton <akpm@linux-foundation.org> > > Date: Thu, 11 Sep 2008 17:02:58 -0700 > > Subject: [PATCH] lock debug: sit tight when we are already in a panic > > > > in: > > > > > http://bugzilla.kernel.org/show_bug.cgi?id=11543 > > > > The panic code called the kexec code which called mutex_trylock() which > > called spin_lock_mutex() which then stupidly went and blurted a load of > > debug stuff because of in_interrupt(). > > > > Keep the lock debug code from escallating an already crappy situation. > > > > Signed-off-by: Ingo Molnar <mingo@elte.hu> > > --- > > include/linux/debug_locks.h | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h > > index 4aaa4af..096476f 100644 > > --- a/include/linux/debug_locks.h > > +++ b/include/linux/debug_locks.h > > @@ -17,7 +17,7 @@ extern int debug_locks_off(void); > > ({ \ > > int __ret = 0; \ > > \ > > - if (unlikely(c)) { \ > > + if (!oops_in_progress && unlikely(c)) { \ > > if (debug_locks_off() && !debug_locks_silent) \ > > WARN_ON(1); \ > > __ret = 1; \ > [ 0.000000] Initializing cgroup subsys cpuset > [ 0.000000] Linux version 2.6.27-rc4-21704-gd25e26b (root@ipp000) (gcc > version 4.1.2 (Gentoo 4.1.2 p1.1)) #1 SMP Tue Sep 2 16:05:45 HST 2008 > [ 0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc > nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8 > [ 0.000000] KERNEL supported cpus: > [ 0.000000] Intel GenuineIntel > [ 0.000000] AMD AuthenticAMD > [ 0.000000] Centaur CentaurHauls > [ 0.000000] BIOS-provided physical RAM map: > [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) > [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) > [ 0.000000] BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved) > [ 0.000000] BIOS-e820: 0000000000100000 - 00000000cff60000 (usable) > [ 0.000000] BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data) > [ 0.000000] BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS) > [ 0.000000] BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved) > [ 0.000000] BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) > [ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) > [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) > [ 0.000000] BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved) > [ 0.000000] BIOS-e820: 0000000100000000 - 0000000230000000 (usable) > [ 0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff > [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new > 0x7010600070106 > [ 0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff > [ 0.000000] init_memory_mapping > [ 0.000000] 0000000000 - 00cfe00000 page 2M > [ 0.000000] 00cfe00000 - 00cff60000 page 4k > [ 0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000 > [ 0.000000] last_map_addr: cff60000 end: cff60000 > [ 0.000000] init_memory_mapping > [ 0.000000] 0100000000 - 0230000000 page 2M > [ 0.000000] kernel direct mapping tables up to 230000000 @ c000-16000 > [ 0.000000] last_map_addr: 230000000 end: 230000000 > [ 0.000000] RAMDISK: 37e0f000 - 37fef7dc > [ 0.000000] DMI present. > [ 0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD ) > [ 0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD XSDT 6040000 LTP > 0) > [ 0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S 6040000 PTL_ > F4240) > [ 0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA CK8 6040000 MSFT > 3000000) > [ 0.000000] ACPI: FACS CFF6AFC0, 0040 > [ 0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD POWERNOW 6040000 AMD > 1) > [ 0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD HAMMER 6040000 AMD > 1) > [ 0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD $UCRTBL$ 6040000 PTL > 1) > [ 0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD MCFG 6040000 LTP > 0) > [ 0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD HPETTBL 6040000 LTP > 1) > [ 0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD APIC 6040000 LTP > 0) > [ 0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD $SBFTBL$ 6040000 LTP > 1) > [ 0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0 > [ 0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0 > [ 0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1 > [ 0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1 > [ 0.000000] SRAT: Node 0 PXM 0 0-a0000 > [ 0.000000] SRAT: Node 0 PXM 0 100000-d0000000 > [ 0.000000] SRAT: Node 0 PXM 0 100000000-130000000 > [ 0.000000] SRAT: Node 1 PXM 1 130000000-230000000 > [ 0.000000] NUMA: Allocated memnodemap from 11000 - 15680 > [ 0.000000] NUMA: Using 20 for the hash shift. > [ 0.000000] Bootmem setup node 0 0000000000000000-0000000130000000 > [ 0.000000] NODE_DATA [0000000000015680 - 000000000002067f] > [ 0.000000] bootmap [0000000000021000 - 0000000000046fff] pages 26 > [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000] > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> > [0000000000 - 0000001000] > [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> > [0000006000 - 0000008000] > [ 0.000000] #2 [0000200000 - 0000f96890] TEXT DATA BSS ==> > [0000200000 - 0000f96890] > [ 0.000000] #3 [0037e0f000 - 0037fef7dc] RAMDISK ==> > [0037e0f000 - 0037fef7dc] > [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved ==> > [000009dc00 - 0000100000] > [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE ==> > [0000008000 - 000000c000] > [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE ==> > [000000c000 - 0000011000] > [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP ==> > [0000011000 - 0000015680] > [ 0.000000] Bootmem setup node 1 0000000130000000-0000000230000000 > [ 0.000000] NODE_DATA [0000000130000000 - 000000013000afff] > [ 0.000000] bootmap [000000013000b000 - 000000013002afff] pages 20 > [ 0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000] > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page > [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE > [ 0.000000] #2 [0000200000 - 0000f96890] TEXT DATA BSS > [ 0.000000] #3 [0037e0f000 - 0037fef7dc] RAMDISK > [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved > [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE > [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE > [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP > [ 0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600 > [ 0.000000] [ffffe20000000000-ffffe200071fffff] PMD -> > [ffff880028200000-ffff88002e1fffff] on node 0 > [ 0.000000] [ffffe20007200000-ffffe2000d1fffff] PMD -> > [ffff880130200000-ffff8801361fffff] on node 1 > [ 0.000000] Zone PFN ranges: > [ 0.000000] DMA 0x00000000 -> 0x00001000 > [ 0.000000] DMA32 0x00001000 -> 0x00100000 > [ 0.000000] Normal 0x00100000 -> 0x00230000 > [ 0.000000] Movable zone start PFN for each node > [ 0.000000] early_node_map[4] active PFN ranges > [ 0.000000] 0: 0x00000000 -> 0x0000009d > [ 0.000000] 0: 0x00000100 -> 0x000cff60 > [ 0.000000] 0: 0x00100000 -> 0x00130000 > [ 0.000000] 1: 0x00130000 -> 0x00230000 > [ 0.000000] On node 0 totalpages: 1048317 > [ 0.000000] DMA zone: 307 pages, LIFO batch:0 > [ 0.000000] DMA32 zone: 823232 pages, LIFO batch:31 > [ 0.000000] Normal zone: 192000 pages, LIFO batch:31 > [ 0.000000] On node 1 totalpages: 1048576 > [ 0.000000] Normal zone: 1024000 pages, LIFO batch:31 > [ 0.000000] Detected use of extended apic ids on hypertransport bus > [ 0.000000] Detected use of extended apic ids on hypertransport bus > [ 0.000000] ACPI: PM-Timer IO Port: 0x1008 > [ 0.000000] ACPI: Local APIC address 0xfee00000 > [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) > [ 0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > [ 0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23 > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > [ 0.000000] ACPI: IRQ0 used by override. > [ 0.000000] ACPI: IRQ2 used by override. > [ 0.000000] ACPI: IRQ9 used by override. > [ 0.000000] Setting APIC routing to flat > [ 0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000 > [ 0.000000] Using ACPI (MADT) for SMP configuration information > [ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs > [ 0.000000] PM: Registered nosave memory: 000000000009d000 - > 000000000009e000 > [ 0.000000] PM: Registered nosave memory: 000000000009e000 - > 00000000000a0000 > [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - > 00000000000ce000 > [ 0.000000] PM: Registered nosave memory: 00000000000ce000 - > 0000000000100000 > [ 0.000000] PM: Registered nosave memory: 00000000cff60000 - > 00000000cff6a000 > [ 0.000000] PM: Registered nosave memory: 00000000cff6a000 - > 00000000cff80000 > [ 0.000000] PM: Registered nosave memory: 00000000cff80000 - > 00000000d0000000 > [ 0.000000] PM: Registered nosave memory: 00000000d0000000 - > 00000000e0000000 > [ 0.000000] PM: Registered nosave memory: 00000000e0000000 - > 00000000f0000000 > [ 0.000000] PM: Registered nosave memory: 00000000f0000000 - > 00000000fec00000 > [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - > 00000000fec10000 > [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - > 00000000fee00000 > [ 0.000000] PM: Registered nosave memory: 00000000fee00000 - > 00000000fee01000 > [ 0.000000] PM: Registered nosave memory: 00000000fee01000 - > 00000000fff80000 > [ 0.000000] PM: Registered nosave memory: 00000000fff80000 - > 0000000100000000 > [ 0.000000] Allocating PCI resources starting at d1000000 (gap: > d0000000:10000000) > [ 0.000000] PERCPU: Allocating 51872 bytes of per cpu data > [ 0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2 > [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total > pages: 2039539 > [ 0.000000] Policy zone: Normal > [ 0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 > init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8 > [ 0.000000] Initializing CPU#0 > [ 0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes) > [ 0.000000] Extended CMOS year: 2000 > [ 0.000000] TSC calibrated against PM_TIMER > [ 0.000000] Detected 2800.009 MHz processor. > [ 0.003333] spurious 8259A interrupt: IRQ7. > [ 0.003333] Console: colour VGA+ 80x25 > [ 0.003333] console [tty0] enabled > [ 0.003333] console [ttyS0] enabled > [ 0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., > Ingo Molnar > [ 0.003333] ... MAX_LOCKDEP_SUBCLASSES: 8 > [ 0.003333] ... MAX_LOCK_DEPTH: 48 > [ 0.003333] ... MAX_LOCKDEP_KEYS: 8191 > [ 0.003333] ... CLASSHASH_SIZE: 4096 > [ 0.003333] ... MAX_LOCKDEP_ENTRIES: 8192 > [ 0.003333] ... MAX_LOCKDEP_CHAINS: 16384 > [ 0.003333] ... CHAINHASH_SIZE: 8192 > [ 0.003333] memory used by lock dependency info: 3839 kB > [ 0.003333] per task-struct memory footprint: 1920 bytes > [ 0.003333] Checking aperture... > [ 0.003333] No AGP bridge found > [ 0.003333] Node 0: aperture @ 0 size 32 MB > [ 0.003333] Your BIOS doesn't leave a aperture memory hole > [ 0.003333] Please enable the IOMMU option in the BIOS setup > [ 0.003333] This costs you 64 MB of RAM > [ 0.003333] Mapping aperture over 65536 KB of RAM @ 20000000 > [ 0.003333] PM: Registered nosave memory: 0000000020000000 - > 0000000024000000 > [ 0.003333] Memory: 8108304k/9175040k available (3896k kernel code, > 279268k reserved, 2243k data, 676k init) > [ 0.003333] CPA: page pool initialized 1 of 1 pages preallocated > [ 0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, > CPUs=4, Nodes=2 > [ 0.003333] hpet clockevent registered > [ 0.003333] Calibrating delay loop (skipped), value calculated using timer > frequency.. 5602.35 BogoMIPS (lpj=9333363) > [ 0.006698] Security Framework initialized > [ 0.010737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 > bytes) > [ 0.018937] Inode-cache hash table entries: 524288 (order: 10, 4194304 > bytes) > [ 0.025421] Mount-cache hash table entries: 256 > [ 0.026974] Initializing cgroup subsys ns > [ 0.030010] Initializing cgroup subsys cpuacct > [ 0.033335] Initializing cgroup subsys memory > [ 0.036674] Initializing cgroup subsys devices > [ 0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > [ 0.043332] CPU: L2 Cache: 1024K (64 bytes/line) > [ 0.046666] CPU 0/0 -> Node 0 > [ 0.049999] tseg: 00cff80000 > [ 0.050000] CPU: Physical Processor ID: 0 > [ 0.053331] CPU: Processor Core ID: 0 > [ 0.056670] using C1E aware idle routine > [ 0.060018] ACPI: Core revision 20080609 > [ 0.070530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 > [ 0.108872] activating NMI Watchdog ... done. > [ 0.109994] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > [ 0.116660] Using local APIC timer interrupts. > [ 0.123326] APIC timer calibration result 12500076 > [ 0.123328] Detected 12.500 MHz APIC timer. > [ 0.126660] APIC timer registered as dummy, due to nmi_watchdog=1! > [ 0.130154] lockdep: fixing up alternatives. > [ 0.133347] Booting processor 1/1 ip 6000 > [ 0.003333] Initializing CPU#1 > [ 0.003333] Calibrating delay using timer specific routine.. 5602.36 > BogoMIPS (lpj=9333385) > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > [ 0.003333] CPU 1/1 -> Node 0 > [ 0.003333] CPU: Physical Processor ID: 0 > [ 0.003333] CPU: Processor Core ID: 1 > [ 0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new > 0x7010600070106 > [ 0.230372] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > [ 0.238145] lockdep: fixing up alternatives. > [ 0.239999] Booting processor 2/2 ip 6000 > [ 0.003333] Initializing CPU#2 > [ 0.003333] Calibrating delay using timer specific routine.. 5602.44 > BogoMIPS (lpj=9333513) > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > [ 0.003333] CPU 2/2 -> Node 1 > [ 0.003333] CPU: Physical Processor ID: 1 > [ 0.003333] CPU: Processor Core ID: 0 > [ 0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new > 0x7010600070106 > [ 0.340530] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > [ 0.348138] lockdep: fixing up alternatives. > [ 0.349997] Booting processor 3/3 ip 6000 > [ 0.003333] Initializing CPU#3 > [ 0.003333] Calibrating delay using timer specific routine.. 5602.48 > BogoMIPS (lpj=9333575) > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > [ 0.003333] CPU 3/3 -> Node 1 > [ 0.003333] CPU: Physical Processor ID: 1 > [ 0.003333] CPU: Processor Core ID: 1 > [ 0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new > 0x7010600070106 > [ 0.447118] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > [ 0.454734] Brought up 4 CPUs > [ 0.456640] Total of 4 processors activated (22409.64 BogoMIPS). > [ 0.460033] Testing NMI watchdog ... > [ 0.532432] WARNING: CPU#0: NMI appears to be stuck (0->0)! > [ 0.533301] Please report this to bugzilla.kernel.org, > [ 0.536635] and attach the output of the 'dmesg' command. > [ 0.539968] > [ 0.543302] WARNING: CPU#1: NMI appears to be stuck (0->0)! > [ 0.546635] Please report this to bugzilla.kernel.org, > [ 0.549968] and attach the output of the 'dmesg' command. > [ 0.553301] > [ 0.554980] WARNING: CPU#2: NMI appears to be stuck (0->0)! > [ 0.556635] Please report this to bugzilla.kernel.org, > [ 0.559968] and attach the output of the 'dmesg' command. > [ 0.563301] > [ 0.566635] WARNING: CPU#3: NMI appears to be stuck (0->0)! > [ 0.569968] Please report this to bugzilla.kernel.org, > [ 0.573301] and attach the output of the 'dmesg' command. > [ 0.577183] net_namespace: 968 bytes > [ 0.580160] xor: automatically using best checksumming function: > generic_sse > [ 0.599969] generic_sse: 8630.400 MB/sec > [ 0.603302] xor: using function: generic_sse (8630.400 MB/sec) > [ 0.606692] NET: Registered protocol family 16 > [ 0.610254] No dock devices found. > [ 0.613503] node 0 link 2: io port [1000, 3fff] > [ 0.613506] TOM: 00000000d0000000 aka 3328M > [ 0.616636] node 0 link 2: mmio [d0000000, dfffffff] > [ 0.616638] node 0 link 2: mmio [a0000, bffff] > [ 0.616640] node 0 link 2: mmio [e0000000, e7ffffff] > [ 0.616643] TOM2: 0000000230000000 aka 8960M > [ 0.619969] bus: [00,ff] on node 0 link 2 > [ 0.619970] bus: 00 index 0 io port: [0, ffff] > [ 0.619972] bus: 00 index 1 mmio: [d0000000, ffffffff] > [ 0.619974] bus: 00 index 2 mmio: [a0000, bffff] > [ 0.619975] bus: 00 index 3 mmio: [230000000, fcffffffff] > [ 0.619989] ACPI: bus type pci registered > [ 0.623448] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4 > [ 0.626636] PCI: MCFG area at e0000000 reserved in E820 > [ 0.630065] PCI: Using MMCONFIG at e0000000 - e04fffff > [ 0.633302] PCI: Using configuration type 1 for base access > [ 0.643980] ACPI: EC: Look up EC in DSDT > [ 0.648409] ACPI: Interpreter enabled > [ 0.649969] ACPI: (supports S0 S1 S3 S4 S5) > [ 0.656635] ACPI: Using IOAPIC for interrupt routing > [ 0.670034] ACPI: PCI Root Bridge [PCI0] (0000:00) > [ 0.673616] pci 0000:00:01.1: PME# supported from D3hot D3cold > [ 0.676644] pci 0000:00:01.1: PME# disabled > [ 0.680009] pci 0000:00:02.0: supports D1 > [ 0.680010] pci 0000:00:02.0: supports D2 > [ 0.680012] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.683303] pci 0000:00:02.0: PME# disabled > [ 0.686672] pci 0000:00:02.1: supports D1 > [ 0.686673] pci 0000:00:02.1: supports D2 > [ 0.686675] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.689969] pci 0000:00:02.1: PME# disabled > [ 0.693535] pci 0000:00:08.0: supports D1 > [ 0.693537] pci 0000:00:08.0: supports D2 > [ 0.693538] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.696637] pci 0000:00:08.0: PME# disabled > [ 0.700017] pci 0000:00:09.0: supports D1 > [ 0.700019] pci 0000:00:09.0: supports D2 > [ 0.700020] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.703304] pci 0000:00:09.0: PME# disabled > [ 0.706664] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.709969] pci 0000:00:0a.0: PME# disabled > [ 0.713331] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.716636] pci 0000:00:0d.0: PME# disabled > [ 0.719997] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold > [ 0.723302] pci 0000:00:0f.0: PME# disabled > [ 0.726878] pci 0000:01:04.0: supports D1 > [ 0.726879] pci 0000:01:04.0: supports D2 > [ 0.726904] pci 0000:00:06.0: transparent bridge > [ 0.729969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff] > [ 0.733303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff] > [ 0.736636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff] > [ 0.740034] pci 0000:02:00.0: supports D1 > [ 0.740064] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff] > [ 0.743388] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] > [ 0.743462] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT] > [ 0.743486] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT] > [ 0.743513] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT] > [ 0.743540] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT] > [ 0.769795] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.777165] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.786790] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.797164] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.808626] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.816220] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.822660] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.830122] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.840505] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.851950] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.858235] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.867922] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.877614] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.887303] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.898166] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 > 19 20 21 22 23) *0, disabled. > [ 0.909241] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.915682] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.921892] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > [ 0.928367] Linux Plug and Play Support v0.97 (c) Adam Belay > [ 0.930030] pnp: PnP ACPI init > [ 0.933313] ACPI: bus type pnp registered > [ 0.942967] pnp: PnP ACPI: found 15 devices > [ 0.943302] ACPI: ACPI bus type pnp unregistered > [ 0.946899] SCSI subsystem initialized > [ 0.950132] libata version 3.00 loaded. > [ 0.953483] usbcore: registered new interface driver usbfs > [ 0.956699] usbcore: registered new interface driver hub > [ 0.960060] usbcore: registered new device driver usb > [ 0.963671] PCI: Using ACPI for IRQ routing > [ 0.980227] PCI-DMA: Disabling AGP. > [ 0.984431] PCI-DMA: aperture base @ 20000000 size 65536 KB > [ 0.986635] PCI-DMA: using GART IOMMU. > [ 0.989971] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture > [ 0.993870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 > [ 1.000341] hpet0: 3 32-bit timers, 25000000 Hz > [ 1.004395] ACPI: RTC can wake from S4 > [ 1.006637] Clockevents: could not switch to one-shot mode:<6>Clockevents: > could not switch to one-shot mode: lapic is not functional. > [ 1.009524] Could not switch to high resolution mode on CPU 3 > [ 1.009528] Clockevents: could not switch to one-shot mode: lapic is not > functional. > [ 1.009531] Could not switch to high resolution mode on CPU 1 > [ 1.009534] Clockevents: could not switch to one-shot mode: lapic is not > functional. > [ 1.009538] Could not switch to high resolution mode on CPU 2 > [ 1.009969] lapic is not functional. > [ 1.056944] Could not switch to high resolution mode on CPU 0 > [ 1.076221] system 00:00: iomem range 0xffc00000-0xffffffff could not be > reserved > [ 1.084054] system 00:00: iomem range 0xfec00000-0xfec00fff could not be > reserved > [ 1.091883] system 00:00: iomem range 0xfee00000-0xfeefffff could not be > reserved > [ 1.099719] system 00:00: iomem range 0xfed00000-0xfed00fff has been > reserved > [ 1.107052] system 00:03: iomem range 0xd0000000-0xd0007fff has been > reserved > [ 1.114383] system 00:04: ioport range 0x1000-0x107f has been reserved > [ 1.121101] system 00:04: ioport range 0x1080-0x10ff has been reserved > [ 1.127817] system 00:04: ioport range 0x1400-0x147f has been reserved > [ 1.134537] system 00:04: ioport range 0x1480-0x14ff has been reserved > [ 1.141258] system 00:04: ioport range 0x1800-0x187f has been reserved > [ 1.147976] system 00:04: ioport range 0x1880-0x18ff has been reserved > [ 1.154698] system 00:04: ioport range 0x2440-0x247f has been reserved > [ 1.161420] system 00:04: ioport range 0x2400-0x243f has been reserved > [ 1.168151] system 00:07: ioport range 0x4d0-0x4d1 has been reserved > [ 1.174695] system 00:07: ioport range 0xc05-0xc06 has been reserved > [ 1.182277] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01 > [ 1.188388] pci 0000:00:06.0: IO window: 0x3000-0x3fff > [ 1.193889] pci 0000:00:06.0: MEM window: 0xd0100000-0xd01fffff > [ 1.200176] pci 0000:00:06.0: PREFETCH window: > 0x000000d8000000-0x000000dfffffff > [ 1.208089] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02 > [ 1.214205] pci 0000:00:0a.0: IO window: disabled > [ 1.219275] pci 0000:00:0a.0: MEM window: 0xd0200000-0xd02fffff > [ 1.225558] pci 0000:00:0a.0: PREFETCH window: > 0x000000d1000000-0x000000d10fffff > [ 1.233474] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03 > [ 1.239598] pci 0000:00:0d.0: IO window: disabled > [ 1.244666] pci 0000:00:0d.0: MEM window: disabled > [ 1.249825] pci 0000:00:0d.0: PREFETCH window: disabled > [ 1.255416] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04 > [ 1.261525] pci 0000:00:0f.0: IO window: disabled > [ 1.266594] pci 0000:00:0f.0: MEM window: disabled > [ 1.271751] pci 0000:00:0f.0: PREFETCH window: disabled > [ 1.277347] pci 0000:00:06.0: setting latency timer to 64 > [ 1.277352] pci 0000:00:0a.0: setting latency timer to 64 > [ 1.277357] pci 0000:00:0d.0: setting latency timer to 64 > [ 1.277363] pci 0000:00:0f.0: setting latency timer to 64 > [ 1.277365] bus: 00 index 0 io port: [0, ffff] > [ 1.282004] bus: 00 index 1 mmio: [0, ffffffffffffffff] > [ 1.292287] bus: 01 index 0 io port: [3000, 3fff] > [ 1.297187] bus: 01 index 1 mmio: [d0100000, d01fffff] > [ 1.302519] bus: 01 index 2 mmio: [d8000000, dfffffff] > [ 1.307849] bus: 01 index 3 io port: [0, ffff] > [ 1.312483] bus: 01 index 4 mmio: [0, ffffffffffffffff] > [ 1.317900] bus: 02 index 0 mmio: [0, 0] > [ 1.322017] bus: 02 index 1 mmio: [d0200000, d02fffff] > [ 1.327349] bus: 02 index 2 mmio: [d1000000, d10fffff] > [ 1.332678] bus: 02 index 3 mmio: [0, 0] > [ 1.336793] bus: 03 index 0 mmio: [0, 0] > [ 1.340910] bus: 03 index 1 mmio: [0, 0] > [ 1.345025] bus: 03 index 2 mmio: [0, 0] > [ 1.349142] bus: 03 index 3 mmio: [0, 0] > [ 1.353256] bus: 04 index 0 mmio: [0, 0] > [ 1.357374] bus: 04 index 1 mmio: [0, 0] > [ 1.361490] bus: 04 index 2 mmio: [0, 0] > [ 1.365609] bus: 04 index 3 mmio: [0, 0] > [ 1.369739] NET: Registered protocol family 2 > [ 1.413137] IP route cache hash table entries: 262144 (order: 9, 2097152 > bytes) > [ 1.423160] TCP established hash table entries: 524288 (order: 11, 8388608 > bytes) > [ 1.436206] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes) > [ 1.445874] TCP: Hash tables configured (established 524288 bind 65536) > [ 1.452716] TCP reno registered > [ 1.466350] NET: Registered protocol family 1 > [ 1.471023] checking if image is initramfs... it is > [ 1.572646] Freeing initrd memory: 1921k freed > [ 1.578336] Simple Boot Flag at 0x37 set to 0x80 > [ 1.585905] audit: initializing netlink socket (disabled) > [ 1.591560] type=2000 audit(1221176947.589:1): initialized > [ 1.602577] HugeTLB registered 2 MB page size, pre-allocated 0 pages > [ 1.613210] VFS: Disk quotas dquot_6.5.1 > [ 1.617471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) > [ 1.625780] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). > [ 1.632809] msgmni has been set to 15840 > [ 1.637191] async_tx: api initialized (async) > [ 1.641747] io scheduler noop registered > [ 1.645871] io scheduler anticipatory registered > [ 1.650694] io scheduler deadline registered > [ 1.655347] io scheduler cfq registered (default) > [ 1.660281] pci 0000:00:00.0: Enabling HT MSI Mapping > [ 1.879582] pci 0000:00:05.0: Enabling HT MSI Mapping > [ 1.884851] pci 0000:00:05.1: Enabling HT MSI Mapping > [ 1.890123] pci 0000:00:05.2: Enabling HT MSI Mapping > [ 1.895393] pci 0000:00:06.0: Enabling HT MSI Mapping > [ 1.900669] pci 0000:00:08.0: Enabling HT MSI Mapping > [ 1.905945] pci 0000:00:09.0: Enabling HT MSI Mapping > [ 1.911213] pci 0000:00:0a.0: Enabling HT MSI Mapping > [ 1.916486] pci 0000:00:0d.0: Enabling HT MSI Mapping > [ 1.921761] pci 0000:00:0f.0: Enabling HT MSI Mapping > [ 1.927029] pci 0000:01:04.0: Boot video device > [ 1.927212] pcieport-driver 0000:00:0a.0: setting latency timer to 64 > [ 1.927241] pcieport-driver 0000:00:0a.0: found MSI capability > [ 1.933324] pci_express 0000:00:0a.0:pcie00: allocate port service > [ 1.933414] pci_express 0000:00:0a.0:pcie01: allocate port service > [ 1.933475] pci_express 0000:00:0a.0:pcie03: allocate port service > [ 1.933576] pcieport-driver 0000:00:0d.0: setting latency timer to 64 > [ 1.933604] pcieport-driver 0000:00:0d.0: found MSI capability > [ 1.939667] pci_express 0000:00:0d.0:pcie00: allocate port service > [ 1.939741] pci_express 0000:00:0d.0:pcie01: allocate port service > [ 1.939801] pci_express 0000:00:0d.0:pcie03: allocate port service > [ 1.939898] pcieport-driver 0000:00:0f.0: setting latency timer to 64 > [ 1.939925] pcieport-driver 0000:00:0f.0: found MSI capability > [ 1.945977] pci_express 0000:00:0f.0:pcie00: allocate port service > [ 1.946048] pci_express 0000:00:0f.0:pcie01: allocate port service > [ 1.946108] pci_express 0000:00:0f.0:pcie03: allocate port service > [ 1.947934] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no > _OSC support > [ 1.949554] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no > _OSC support > [ 1.951148] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no > _OSC support > [ 1.951606] input: Power Button (FF) as /class/input/input0 > [ 1.957384] ACPI: Power Button (FF) [PWRF] > [ 1.961801] input: Power Button (CM) as /class/input/input1 > [ 1.967565] ACPI: Power Button (CM) [PWRB] > [ 1.972294] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) > [ 1.978363] processor ACPI0007:00: registered as cooling_device0 > [ 1.984576] ACPI: Processor [C000] (supports 2 throttling states) > [ 1.991063] processor ACPI0007:01: registered as cooling_device1 > [ 1.997366] processor ACPI0007:02: registered as cooling_device2 > [ 2.003684] processor ACPI0007:03: registered as cooling_device3 > [ 2.079999] hpet_resources: 0xfed00000 is busy > [ 2.080193] Non-volatile memory driver v1.2 > [ 2.084715] Linux agpgart interface v0.103 > [ 2.089107] Serial: 8250/16550 driver4 ports, IRQ sharing disabled > [ 2.095680] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A > [ 2.102659] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A > [ 2.108789] Floppy drive(s): fd0 is 1.44M > [ 2.130069] FDC 0 is a post-1991 82077 > [ 2.137592] brd: module loaded > [ 2.142041] loop: module loaded > [ 2.145379] tun: Universal TUN/TAP device driver, 1.6 > [ 2.150619] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> > [ 2.157181] console [netcon0] enabled > [ 2.161035] netconsole: network logging started > [ 2.165762] Uniform Multi-Platform E-IDE driver > [ 2.170615] ide_generic: please use "probe_mask=0x3f" module parameter for > probing all legacy ISA IDE ports > [ 2.180715] Probing IDE interface ide0... > [ 3.149712] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive > [ 3.206282] Probing IDE interface ide1... > [ 3.739617] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 > [ 3.744443] ide1 at 0x170-0x177,0x376 on irq 15 > [ 3.751258] hdb: ATAPI 24X CD-ROM drive, 256kB Cache > [ 3.756646] Uniform CD-ROM driver Revision: 3.20 > [ 3.771462] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST > 2006) > [ 3.779263] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST > 2006) > [ 3.786512] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008 > [ 3.792690] Driver 'sd' needs updating - please use bus_type methods > [ 3.799315] Driver 'sr' needs updating - please use bus_type methods > [ 3.806449] sata_nv 0000:00:05.0: version 3.5 > [ 3.806741] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23 > [ 3.812700] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 > (level, low) -> IRQ 23 > [ 3.821311] sata_nv 0000:00:05.0: Using SWNCQ mode > [ 3.826430] sata_nv 0000:00:05.0: setting latency timer to 64 > [ 3.826613] scsi0 : sata_nv > [ 3.829796] scsi1 : sata_nv > [ 3.832986] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq > 23 > [ 3.840314] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq > 23 > [ 4.179871] ata1: SATA link down (SStatus 0 SControl 300) > [ 4.516536] ata2: SATA link down (SStatus 0 SControl 300) > [ 4.522429] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22 > [ 4.528388] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 > (level, low) -> IRQ 22 > [ 4.536997] sata_nv 0000:00:05.1: Using SWNCQ mode > [ 4.542083] sata_nv 0000:00:05.1: setting latency timer to 64 > [ 4.542220] scsi2 : sata_nv > [ 4.545334] scsi3 : sata_nv > [ 4.548512] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq > 22 > [ 4.555840] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq > 22 > [ 4.896535] ata3: SATA link down (SStatus 0 SControl 300) > [ 5.233200] ata4: SATA link down (SStatus 0 SControl 300) > [ 5.239087] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21 > [ 5.245047] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 > (level, low) -> IRQ 21 > [ 5.253654] sata_nv 0000:00:05.2: Using SWNCQ mode > [ 5.258744] sata_nv 0000:00:05.2: setting latency timer to 64 > [ 5.258881] scsi4 : sata_nv > [ 5.261998] scsi5 : sata_nv > [ 5.265174] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq > 21 > [ 5.272504] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq > 21 > [ 5.613197] ata5: SATA link down (SStatus 0 SControl 300) > [ 5.949864] ata6: SATA link down (SStatus 0 SControl 300) > [ 5.955612] pata_amd 0000:00:04.0: version 0.3.10 > [ 5.955640] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region > [0x1f0-0x1f7] > [ 5.963385] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 > (errno=-16) > [ 5.971653] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region > [0x170-0x177] > [ 5.979396] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 > (errno=-16) > [ 5.987656] pata_amd 0000:00:04.0: no available native port > [ 5.993731] Fusion MPT base driver 3.04.07 > [ 5.998021] Copyright (c) 1999-2008 LSI Corporation > [ 6.003109] Fusion MPT SPI Host driver 3.04.07 > [ 6.007833] Fusion MPT SAS Host driver 3.04.07 > [ 6.013164] ieee1394: raw1394: /dev/raw1394 device initialized > [ 6.019695] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20 > [ 6.025650] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 > (level, low) -> IRQ 20 > [ 6.034353] ehci_hcd 0000:00:02.1: setting latency timer to 64 > [ 6.034355] ehci_hcd 0000:00:02.1: EHCI Host Controller > [ 6.040002] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus > number 1 > [ 6.047776] ehci_hcd 0000:00:02.1: debug port 1 > [ 6.052503] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported > [ 6.052521] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000 > [ 6.066176] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 > Dec 2004 > [ 6.074154] usb usb1: configuration #1 chosen from 1 choice > [ 6.080016] hub 1-0:1.0: USB hub found > [ 6.083976] hub 1-0:1.0: 10 ports detected > [ 6.189886] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) > Driver > [ 6.190142] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19 > [ 6.196100] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 > (level, low) -> IRQ 19 > [ 6.204801] ohci_hcd 0000:00:02.0: setting latency timer to 64 > [ 6.204804] ohci_hcd 0000:00:02.0: OHCI Host Controller > [ 6.210376] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus > number 2 > [ 6.218132] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000 > [ 6.278315] usb usb2: configuration #1 chosen from 1 choice > [ 6.284165] hub 2-0:1.0: USB hub found > [ 6.288118] hub 2-0:1.0: 10 ports detected > [ 6.393142] USB Universal Host Controller Interface driver v3.0 > [ 6.399423] usbcore: registered new interface driver usblp > [ 6.405107] Initializing USB Mass Storage driver... > [ 6.410243] usbcore: registered new interface driver usb-storage > [ 6.416443] USB Mass Storage support registered. > [ 6.421406] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 > irq 1,12 > [ 6.679856] serio: i8042 KBD port at 0x60,0x64 irq 1 > [ 6.685238] mice: PS/2 mouse device common for all mice > [ 6.877983] md: linear personality registered for level -1 > [ 6.883663] md: raid0 personality registered for level 0 > [ 6.889166] md: raid1 personality registered for level 1 > [ 6.894669] md: raid10 personality registered for level 10 > [ 6.956180] raid6: int64x1 2587 MB/s > [ 7.016180] raid6: int64x2 3291 MB/s > [ 7.076168] raid6: int64x4 3160 MB/s > [ 7.136168] raid6: int64x8 2360 MB/s > [ 7.196178] raid6: sse2x1 3858 MB/s > [ 7.256175] raid6: sse2x2 5139 MB/s > [ 7.316176] raid6: sse2x4 5262 MB/s > [ 7.320122] raid6: using algorithm sse2x4 (5262 MB/s) > [ 7.325365] md: raid6 personality registered for level 6 > [ 7.335744] md: raid5 personality registered for level 5 > [ 7.341252] md: raid4 personality registered for level 4 > [ 7.346753] md: multipath personality registered for level -4 > [ 7.352960] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: > dm-devel@redhat.com > [ 7.361877] cpuidle: using governor ladder > [ 7.366402] cpuidle: using governor menu > [ 7.370591] usbcore: registered new interface driver usbhid > [ 7.376362] usbhid: v2.6:USB HID core driver > [ 7.380936] oprofile: using NMI interrupt. > [ 7.385695] TCP cubic registered > [ 7.389332] NET: Registered protocol family 10 > [ 7.394985] IPv6 over IPv4 tunneling driver > [ 7.400135] NET: Registered protocol family 17 > [ 7.405122] RPC: Registered udp transport module. > [ 7.410021] RPC: Registered tcp transport module. > [ 7.415002] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 > processors (4 cpu cores) (version 2.20.00) > [ 7.425975] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 > [ 7.431831] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa > [ 7.437687] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc > [ 7.443537] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe > [ 7.449301] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 > [ 7.455149] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 > [ 7.460999] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 > [ 7.467050] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 > [ 7.472929] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa > [ 7.478785] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc > [ 7.484637] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe > [ 7.490411] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 > [ 7.496279] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 > [ 7.502134] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 > [ 7.508739] drivers/rtc/hctosys.c: unable to open rtc device (rtc0) > [ 7.515289] Freeing unused kernel memory: 676k freed > [ 7.520818] Write protecting the kernel read-only data: 5672k > [ 7.742843] Clocksource tsc unstable (delta = -111149082 ns) > [ 8.177545] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18 > [ 8.177654] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, > low) -> IRQ 18 > [ 8.177666] arcmsr 0000:02:00.0: setting latency timer to 64 > [ 8.189529] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31 > [ 8.202850] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 > capable) > [ 8.202853] Driver Version 1.20.00.15 2008/02/27 > [ 8.205018] scsi 6:0:0:0: Direct-Access Areca ARC-1280-VOL#00 R001 > PQ: 0 ANSI: 5 > [ 8.207052] sd 6:0:0:0: [sda] Very big device. Trying to use READ > CAPACITY(16). > [ 8.207117] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors > (10999996 MB) > [ 8.207182] sd 6:0:0:0: [sda] Write Protect is off > [ 8.207188] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08 > [ 8.207298] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, > doesn't support DPO or FUA > [ 8.207560] sd 6:0:0:0: [sda] Very big device. Trying to use READ > CAPACITY(16). > [ 8.207621] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors > (10999996 MB) > [ 8.207685] sd 6:0:0:0: [sda] Write Protect is off > [ 8.207692] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08 > [ 8.207802] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, > doesn't support DPO or FUA > [ 8.207811] sda: sda1 sda2 sda3 sda4 > [ 8.245400] sd 6:0:0:0: [sda] Attached SCSI disk > [ 8.247390] sd 6:0:0:0: Attached scsi generic sg0 type 0 > [ 8.249720] scsi 6:0:16:0: Processor Areca RAID controller > R001 PQ: 0 ANSI: 0 > [ 8.252259] scsi 6:0:16:0: Attached scsi generic sg1 type 3 > [ 8.504889] 3ware Storage Controller device driver for Linux v1.26.02.002. > [ 8.542592] 3ware 9000 Storage Controller device driver for Linux > v2.26.02.011. > [ 8.601009] Adaptec aacraid driver 1.1-5[2456]-ms > [ 8.993029] SGI XFS with ACLs, security attributes, large block/inode > numbers, no debug enabled > [ 8.998828] SGI XFS Quota Management subsystem > [ 9.048982] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI > [ 9.048990] Copyright (c) 1999-2006 Intel Corporation. > [ 12.346739] EXT3-fs: INFO: recovery required on readonly filesystem. > [ 12.346748] EXT3-fs: write access will be enabled during recovery. > [ 13.161202] kjournald starting. Commit interval 5 seconds > [ 13.161240] EXT3-fs: sda3: orphan cleanup on readonly fs > [ 13.161256] ext3_orphan_cleanup: deleting unreferenced inode 2501876 > [ 13.161508] ext3_orphan_cleanup: deleting unreferenced inode 2501875 > [ 13.161529] ext3_orphan_cleanup: deleting unreferenced inode 2501874 > [ 13.161549] ext3_orphan_cleanup: deleting unreferenced inode 2501873 > [ 13.161565] EXT3-fs: sda3: 4 orphan inodes deleted > [ 13.161570] EXT3-fs: recovery complete. > [ 13.162085] EXT3-fs: mounted filesystem with ordered data mode. > [ 15.634222] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440 > [ 15.634257] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400 > [ 15.660203] forcedeth: Reverse Engineered nForce ethernet driver. Version > 0.61. > [ 15.660582] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17 > [ 15.660659] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 > (level, low) -> IRQ 17 > [ 15.660663] forcedeth 0000:00:08.0: setting latency timer to 64 > [ 15.663568] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, addr > 00:e0:81:76:8b:be > [ 15.663571] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq > gbit lnktim msi desc-v3 > [ 15.663876] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16 > [ 15.663898] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 > (level, low) -> IRQ 16 > [ 15.663923] forcedeth 0000:00:09.0: setting latency timer to 64 > [ 15.665255] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, addr > 00:e0:81:76:8b:bf > [ 15.665257] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq > gbit lnktim msi desc-v3 > [ 18.107517] EXT3 FS on sda3, internal journal > [ 18.380032] SMsC 37B787 watchdog component driver 1.1 initialising... > [ 18.381424] smsc37b787_wdt: Timeout set to 60 second(s). > [ 18.381429] smsc37b787_wdt: Watchdog initialized and sleeping > (nowayout=0)... > [ 21.417259] kjournald starting. Commit interval 5 seconds > [ 21.417297] EXT3-fs warning: checktime reached, running e2fsck is > recommended > [ 21.422349] EXT3 FS on sda4, internal journal > [ 21.422359] EXT3-fs: recovery complete. > [ 21.422430] EXT3-fs: mounted filesystem with ordered data mode. > [ 21.501039] Adding 15999992k swap on /dev/sda2. Priority:-1 extents:1 > across:15999992k > [ 42.038582] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state > recovery directory > [ 42.045903] NFSD: starting 90-second grace period > [ 45.396027] eth0: no IPv6 routers present > [ 49.761642] w83627hf: Found W83627HF chip at 0xc00 Reply-To: josh@hoblitt.com In addition to the deadlocks, we still have the watchdog warning: [ 0.460034] Testing NMI watchdog ... [ 0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)! [ 0.533301] Please report this to bugzilla.kernel.org, [ 0.536635] and attach the output of the 'dmesg' command. Perhaps an HPET problem: [ 0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 [ 0.999969] hpet0: 3 32-bit timers, 25000000 Hz [ 1.004396] ACPI: RTC can wake from S4 [ 1.006637] Clockevents: could not switch to one-shot mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional. [ 1.009844] Could not switch to high resolution mode on CPU 3 [ 1.009848] Clockevents: could not switch to one-shot mode: lapic is not functional. [ 1.009852] Could not switch to high resolution mode on CPU 2 [ 1.009855] Clockevents: could not switch to one-shot mode: lapic is not functional. [ 1.009858] Could not switch to high resolution mode on CPU 1 [ 1.009969] lapic is not functional. [ 1.056944] Could not switch to high resolution mode on CPU 0 And a failure to create a /dev/rtc[0] device with udev 115 or 119 and this entry in the dmesg. [ 7.498900] drivers/rtc/hctosys.c: unable to open rtc device (rtc0) -J -- On Mon, Sep 15, 2008 at 11:06:35AM -1000, Joshua Hoblitt wrote: > Since Friday 2 different machines have experienced crashes. One was a total > deadlock with no response on the console. The other one reported the trace > below on the console and stopped responding to ssh but I was able to loging > via > the serial console and reboot the system. This particular system has had a > number of "odd" kernel traces over the last year and I'm starting to actually > wonder if it may have a bad DIMM in it as occasionally the failure mode seems > to be different then the deadlocks/etc. we see in the other 15 nodes with > identical hardware. > > [30712.654542] general protection fault: 0000 [1] SMP > <Sep/12 09:25 pm>[30712.657678] CPU 3 > <Sep/12 09:25 pm>[30712.657678] Modules linked in: w83627hf hwmon_vid autofs4 > smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs > dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx > scsi_wait_scan > <Sep/12 09:25 pm>[30712.657678] Pid: 1178, comm: rpciod/3 Not tainted > 2.6.27-rc5-22033-gd26acd9-dirty #2 > <Sep/12 09:25 pm>[30712.657678] RIP: 0010:[<ffffffff805ac57d>] > [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8 > <Sep/12 09:25 pm>[30712.657678] RSP: 0018:ffff88012e5d1e08 EFLAGS: 00010206 > <Sep/12 09:25 pm>[30712.657678] RAX: ffffffff807adb48 RBX: ffff880126d61088 > RCX: 0400000000000000 > <Sep/12 09:25 pm>[30712.657678] RDX: ffff88022bcc0380 RSI: ffff88022bcc0000 > RDI: ffff880126d61088 > <Sep/12 09:25 pm>[30712.657678] RBP: ffff88022bc88000 R08: 0000000000000003 > R09: ffff88022e038f10 > <Sep/12 09:25 pm>[30712.657678] R10: 0000000000000001 R11: ffff88012e4a0048 > R12: 0400000000000000 > <Sep/12 09:25 pm>[30712.657678] R13: ffffffff8059f7a8 R14: ffff88022bc88610 > R15: 0000000000000000 > <Sep/12 09:25 pm>[30712.657678] FS: 00007f006fb306f0(0000) > GS:ffff88022fa0d780(0000) knlGS:0000000000000000 > > <Sep/12 09:25 pm>[30712.657678] CS: 0010 DS: 0018 ES: 0018 CR0: > 000000008005003b > <Sep/12 09:25 pm>[30712.657678] CR2: 00000000011e3018 CR3: 00000001f3527000 > CR4:<Sep/12 09:25 pm> > <Sep/12 09:25 pm>[30712.657678] DR0: 0000000000000000 DR1: > 0000000000000000 DR2: 0000000000000000 > <Sep/12 09:25 pm>[30712.657678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 > DR7: 0000000000000400 > <Sep/12 09:25 pm>[30712.657678] Process rpciod/3 (pid: 1178, threadinfo > ffff88012e5d0000, task ffff88012e5acb50) > <Sep/12 09:25 pm>[30712.657678] Stack: ffffffff8059b996 ffff880126d61088 > 00000000fffffff5 ffff880126d61118 > <Sep/12 09:25 pm>[30712.657678] ffffffff8059f7a8 ffffffff808a4b20 > ffffffff80599aa1 ffffffff8059f7a8 > <Sep/12 09:25 pm>[30712.657678] 0000000000000000 ffff880126d61088 > ffffffff8059f5f3 ffff88012e5d1e90 > <Sep/12 09:25 pm>[30712.657678] Call Trace: > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059b996>] ? > xprt_release+0x2c/0x172 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f7a8>] ? > rpc_async_schedule+0x0/0xc > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80599aa1>] ? > call_reserveresult+0x95/0xd1 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f7a8>] ? > rpc_async_schedule+0x0/0xc > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8059f5f3>] ? > __rpc_execute+0x73/0x228 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244394>] ? > run_workqueue+0xed/0x1ed > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024433e>] ? > run_workqueue+0x97/0x1ed > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244ef7>] ? > worker_thread+0xd8/0xe3 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024791a>] ? > autoremove_wake_function+0x0/0x2e > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80244e1f>] ? > worker_thread+0x0/0xe3 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80247805>] ? kthread+0x47/0x76 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff80230223>] ? > schedule_tail+0x27/0x5f > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8020ce09>] ? child_rip+0xa/0x11 > > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8024767c>] ? kthreadd+0x167/0x18c > <Sep/12 09:25 pm>[30712.657678] [<ffffffff802477be>] ? kthread+0x0/0x76 > <Sep/12 09:25 pm>[30712.657678] [<ffffffff8020cdff>] ? child_rip+0x0/0x11 > <Sep/12 09:25 pm>[30712.657678] > <Sep/12 09:25 pm>[30712.657678] > <Sep/12 09:25 pm>[30712.657678] Code: b0 98 00 00 00 48 85 f6 0f 94 c2 48 85 > c9 0f 94 c0 08 c2 0f 85 94 00 00 00 48 8b 47 38 8b 50 28 48 c1 e2 07 48 8d 14 > 16 48 ff 02 <48> 63 81 50 01 00 00 48 01 42 08 0f b7 87 e0 00 00 00 48 01 42 > <Sep/12 09:25 pm>[30712.657678] RIP [<ffffffff805ac57d>] > rpc_count_iostats+0x35/0xb8 > <Sep/12 09:25 pm>[30712.657678] RSP <ffff88012e5d1e08> > <Sep/12 09:25 pm>[30712.951578] ---[ end trace e646bef2de7f0b20 ]--- > > -J > > -- > On Fri, Sep 12, 2008 at 02:13:06PM -1000, Joshua Hoblitt wrote: > > I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET > > patch Thomas pointed me at. I'll let it roast on these 14 machine is > > production over the weekend to see if we get another panic. > > > > I'm attaching the dmesg from this kernel. We're still getting the NMI > > watchdog warning and the rtc is [still] hosed (I think it was last > > working around -rc3). > > > > -J > > > > -- > > On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote: > > > > > > * Andrew Morton <akpm@linux-foundation.org> wrote: > > > > > > > I can't work out who called panic(), nor why. > > > > > > > > The panic code called the kexec code which called mutex_trylock() > > > > which called spin_lock_mutex() which then stupidly went and blurted a > > > > load of debug stuff because of in_interrupt(). > > > > > > agreed - applied your fix in the form below to tip/master - thanks > > > Andrew. > > > > > > J, you might want to try tip/master, it includes all known fixes for > > > this area and this debug improvement as well. You can pick it up via: > > > > > > http://people.redhat.com/mingo/tip.git/README > > > > > > Ingo > > > > > > ----------> > > > >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001 > > > From: Andrew Morton <akpm@linux-foundation.org> > > > Date: Thu, 11 Sep 2008 17:02:58 -0700 > > > Subject: [PATCH] lock debug: sit tight when we are already in a panic > > > > > > in: > > > > > > > http://bugzilla.kernel.org/show_bug.cgi?id=11543 > > > > > > The panic code called the kexec code which called mutex_trylock() which > > > called spin_lock_mutex() which then stupidly went and blurted a load of > > > debug stuff because of in_interrupt(). > > > > > > Keep the lock debug code from escallating an already crappy situation. > > > > > > Signed-off-by: Ingo Molnar <mingo@elte.hu> > > > --- > > > include/linux/debug_locks.h | 2 +- > > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > > > diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h > > > index 4aaa4af..096476f 100644 > > > --- a/include/linux/debug_locks.h > > > +++ b/include/linux/debug_locks.h > > > @@ -17,7 +17,7 @@ extern int debug_locks_off(void); > > > ({ > \ > > > int __ret = 0; \ > > > \ > > > - if (unlikely(c)) { \ > > > + if (!oops_in_progress && unlikely(c)) { \ > > > if (debug_locks_off() && !debug_locks_silent) \ > > > WARN_ON(1); \ > > > __ret = 1; \ > > > [ 0.000000] Initializing cgroup subsys cpuset > > [ 0.000000] Linux version 2.6.27-rc4-21704-gd25e26b (root@ipp000) (gcc > version 4.1.2 (Gentoo 4.1.2 p1.1)) #1 SMP Tue Sep 2 16:05:45 HST 2008 > > [ 0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 > init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8 > > [ 0.000000] KERNEL supported cpus: > > [ 0.000000] Intel GenuineIntel > > [ 0.000000] AMD AuthenticAMD > > [ 0.000000] Centaur CentaurHauls > > [ 0.000000] BIOS-provided physical RAM map: > > [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) > > [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) > > [ 0.000000] BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved) > > [ 0.000000] BIOS-e820: 0000000000100000 - 00000000cff60000 (usable) > > [ 0.000000] BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data) > > [ 0.000000] BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS) > > [ 0.000000] BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved) > > [ 0.000000] BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) > > [ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) > > [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) > > [ 0.000000] BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved) > > [ 0.000000] BIOS-e820: 0000000100000000 - 0000000230000000 (usable) > > [ 0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff > > [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new > 0x7010600070106 > > [ 0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff > > [ 0.000000] init_memory_mapping > > [ 0.000000] 0000000000 - 00cfe00000 page 2M > > [ 0.000000] 00cfe00000 - 00cff60000 page 4k > > [ 0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000 > > [ 0.000000] last_map_addr: cff60000 end: cff60000 > > [ 0.000000] init_memory_mapping > > [ 0.000000] 0100000000 - 0230000000 page 2M > > [ 0.000000] kernel direct mapping tables up to 230000000 @ c000-16000 > > [ 0.000000] last_map_addr: 230000000 end: 230000000 > > [ 0.000000] RAMDISK: 37e0f000 - 37fef7dc > > [ 0.000000] DMI present. > > [ 0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD ) > > [ 0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD XSDT > 6040000 LTP 0) > > [ 0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S 6040000 PTL_ > F4240) > > [ 0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA CK8 6040000 MSFT > 3000000) > > [ 0.000000] ACPI: FACS CFF6AFC0, 0040 > > [ 0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD POWERNOW 6040000 AMD > 1) > > [ 0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD HAMMER 6040000 AMD > 1) > > [ 0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD $UCRTBL$ 6040000 PTL > 1) > > [ 0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD MCFG 6040000 LTP > 0) > > [ 0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD HPETTBL 6040000 LTP > 1) > > [ 0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD APIC > 6040000 LTP 0) > > [ 0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD $SBFTBL$ 6040000 LTP > 1) > > [ 0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0 > > [ 0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0 > > [ 0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1 > > [ 0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1 > > [ 0.000000] SRAT: Node 0 PXM 0 0-a0000 > > [ 0.000000] SRAT: Node 0 PXM 0 100000-d0000000 > > [ 0.000000] SRAT: Node 0 PXM 0 100000000-130000000 > > [ 0.000000] SRAT: Node 1 PXM 1 130000000-230000000 > > [ 0.000000] NUMA: Allocated memnodemap from 11000 - 15680 > > [ 0.000000] NUMA: Using 20 for the hash shift. > > [ 0.000000] Bootmem setup node 0 0000000000000000-0000000130000000 > > [ 0.000000] NODE_DATA [0000000000015680 - 000000000002067f] > > [ 0.000000] bootmap [0000000000021000 - 0000000000046fff] pages 26 > > [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000] > > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> > [0000000000 - 0000001000] > > [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> > [0000006000 - 0000008000] > > [ 0.000000] #2 [0000200000 - 0000f96890] TEXT DATA BSS ==> > [0000200000 - 0000f96890] > > [ 0.000000] #3 [0037e0f000 - 0037fef7dc] RAMDISK ==> > [0037e0f000 - 0037fef7dc] > > [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved ==> > [000009dc00 - 0000100000] > > [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE ==> > [0000008000 - 000000c000] > > [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE ==> > [000000c000 - 0000011000] > > [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP ==> > [0000011000 - 0000015680] > > [ 0.000000] Bootmem setup node 1 0000000130000000-0000000230000000 > > [ 0.000000] NODE_DATA [0000000130000000 - 000000013000afff] > > [ 0.000000] bootmap [000000013000b000 - 000000013002afff] pages 20 > > [ 0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000] > > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page > > [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE > > [ 0.000000] #2 [0000200000 - 0000f96890] TEXT DATA BSS > > [ 0.000000] #3 [0037e0f000 - 0037fef7dc] RAMDISK > > [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved > > [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE > > [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE > > [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP > > [ 0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600 > > [ 0.000000] [ffffe20000000000-ffffe200071fffff] PMD -> > [ffff880028200000-ffff88002e1fffff] on node 0 > > [ 0.000000] [ffffe20007200000-ffffe2000d1fffff] PMD -> > [ffff880130200000-ffff8801361fffff] on node 1 > > [ 0.000000] Zone PFN ranges: > > [ 0.000000] DMA 0x00000000 -> 0x00001000 > > [ 0.000000] DMA32 0x00001000 -> 0x00100000 > > [ 0.000000] Normal 0x00100000 -> 0x00230000 > > [ 0.000000] Movable zone start PFN for each node > > [ 0.000000] early_node_map[4] active PFN ranges > > [ 0.000000] 0: 0x00000000 -> 0x0000009d > > [ 0.000000] 0: 0x00000100 -> 0x000cff60 > > [ 0.000000] 0: 0x00100000 -> 0x00130000 > > [ 0.000000] 1: 0x00130000 -> 0x00230000 > > [ 0.000000] On node 0 totalpages: 1048317 > > [ 0.000000] DMA zone: 307 pages, LIFO batch:0 > > [ 0.000000] DMA32 zone: 823232 pages, LIFO batch:31 > > [ 0.000000] Normal zone: 192000 pages, LIFO batch:31 > > [ 0.000000] On node 1 totalpages: 1048576 > > [ 0.000000] Normal zone: 1024000 pages, LIFO batch:31 > > [ 0.000000] Detected use of extended apic ids on hypertransport bus > > [ 0.000000] Detected use of extended apic ids on hypertransport bus > > [ 0.000000] ACPI: PM-Timer IO Port: 0x1008 > > [ 0.000000] ACPI: Local APIC address 0xfee00000 > > [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) > > [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) > > [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) > > [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) > > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) > > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) > > [ 0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > > [ 0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI > 0-23 > > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) > > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) > > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > > [ 0.000000] ACPI: IRQ0 used by override. > > [ 0.000000] ACPI: IRQ2 used by override. > > [ 0.000000] ACPI: IRQ9 used by override. > > [ 0.000000] Setting APIC routing to flat > > [ 0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000 > > [ 0.000000] Using ACPI (MADT) for SMP configuration information > > [ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs > > [ 0.000000] PM: Registered nosave memory: 000000000009d000 - > 000000000009e000 > > [ 0.000000] PM: Registered nosave memory: 000000000009e000 - > 00000000000a0000 > > [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - > 00000000000ce000 > > [ 0.000000] PM: Registered nosave memory: 00000000000ce000 - > 0000000000100000 > > [ 0.000000] PM: Registered nosave memory: 00000000cff60000 - > 00000000cff6a000 > > [ 0.000000] PM: Registered nosave memory: 00000000cff6a000 - > 00000000cff80000 > > [ 0.000000] PM: Registered nosave memory: 00000000cff80000 - > 00000000d0000000 > > [ 0.000000] PM: Registered nosave memory: 00000000d0000000 - > 00000000e0000000 > > [ 0.000000] PM: Registered nosave memory: 00000000e0000000 - > 00000000f0000000 > > [ 0.000000] PM: Registered nosave memory: 00000000f0000000 - > 00000000fec00000 > > [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - > 00000000fec10000 > > [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - > 00000000fee00000 > > [ 0.000000] PM: Registered nosave memory: 00000000fee00000 - > 00000000fee01000 > > [ 0.000000] PM: Registered nosave memory: 00000000fee01000 - > 00000000fff80000 > > [ 0.000000] PM: Registered nosave memory: 00000000fff80000 - > 0000000100000000 > > [ 0.000000] Allocating PCI resources starting at d1000000 (gap: > d0000000:10000000) > > [ 0.000000] PERCPU: Allocating 51872 bytes of per cpu data > > [ 0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2 > > [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. > Total pages: 2039539 > > [ 0.000000] Policy zone: Normal > > [ 0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 > init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8 > > [ 0.000000] Initializing CPU#0 > > [ 0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes) > > [ 0.000000] Extended CMOS year: 2000 > > [ 0.000000] TSC calibrated against PM_TIMER > > [ 0.000000] Detected 2800.009 MHz processor. > > [ 0.003333] spurious 8259A interrupt: IRQ7. > > [ 0.003333] Console: colour VGA+ 80x25 > > [ 0.003333] console [tty0] enabled > > [ 0.003333] console [ttyS0] enabled > > [ 0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., > Ingo Molnar > > [ 0.003333] ... MAX_LOCKDEP_SUBCLASSES: 8 > > [ 0.003333] ... MAX_LOCK_DEPTH: 48 > > [ 0.003333] ... MAX_LOCKDEP_KEYS: 8191 > > [ 0.003333] ... CLASSHASH_SIZE: 4096 > > [ 0.003333] ... MAX_LOCKDEP_ENTRIES: 8192 > > [ 0.003333] ... MAX_LOCKDEP_CHAINS: 16384 > > [ 0.003333] ... CHAINHASH_SIZE: 8192 > > [ 0.003333] memory used by lock dependency info: 3839 kB > > [ 0.003333] per task-struct memory footprint: 1920 bytes > > [ 0.003333] Checking aperture... > > [ 0.003333] No AGP bridge found > > [ 0.003333] Node 0: aperture @ 0 size 32 MB > > [ 0.003333] Your BIOS doesn't leave a aperture memory hole > > [ 0.003333] Please enable the IOMMU option in the BIOS setup > > [ 0.003333] This costs you 64 MB of RAM > > [ 0.003333] Mapping aperture over 65536 KB of RAM @ 20000000 > > [ 0.003333] PM: Registered nosave memory: 0000000020000000 - > 0000000024000000 > > [ 0.003333] Memory: 8108304k/9175040k available (3896k kernel code, > 279268k reserved, 2243k data, 676k init) > > [ 0.003333] CPA: page pool initialized 1 of 1 pages preallocated > > [ 0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, > CPUs=4, Nodes=2 > > [ 0.003333] hpet clockevent registered > > [ 0.003333] Calibrating delay loop (skipped), value calculated using > timer frequency.. 5602.35 BogoMIPS (lpj=9333363) > > [ 0.006698] Security Framework initialized > > [ 0.010737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 > bytes) > > [ 0.018937] Inode-cache hash table entries: 524288 (order: 10, 4194304 > bytes) > > [ 0.025421] Mount-cache hash table entries: 256 > > [ 0.026974] Initializing cgroup subsys ns > > [ 0.030010] Initializing cgroup subsys cpuacct > > [ 0.033335] Initializing cgroup subsys memory > > [ 0.036674] Initializing cgroup subsys devices > > [ 0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > > [ 0.043332] CPU: L2 Cache: 1024K (64 bytes/line) > > [ 0.046666] CPU 0/0 -> Node 0 > > [ 0.049999] tseg: 00cff80000 > > [ 0.050000] CPU: Physical Processor ID: 0 > > [ 0.053331] CPU: Processor Core ID: 0 > > [ 0.056670] using C1E aware idle routine > > [ 0.060018] ACPI: Core revision 20080609 > > [ 0.070530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 > > [ 0.108872] activating NMI Watchdog ... done. > > [ 0.109994] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > > [ 0.116660] Using local APIC timer interrupts. > > [ 0.123326] APIC timer calibration result 12500076 > > [ 0.123328] Detected 12.500 MHz APIC timer. > > [ 0.126660] APIC timer registered as dummy, due to nmi_watchdog=1! > > [ 0.130154] lockdep: fixing up alternatives. > > [ 0.133347] Booting processor 1/1 ip 6000 > > [ 0.003333] Initializing CPU#1 > > [ 0.003333] Calibrating delay using timer specific routine.. 5602.36 > BogoMIPS (lpj=9333385) > > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > > [ 0.003333] CPU 1/1 -> Node 0 > > [ 0.003333] CPU: Physical Processor ID: 0 > > [ 0.003333] CPU: Processor Core ID: 1 > > [ 0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new > 0x7010600070106 > > [ 0.230372] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > > [ 0.238145] lockdep: fixing up alternatives. > > [ 0.239999] Booting processor 2/2 ip 6000 > > [ 0.003333] Initializing CPU#2 > > [ 0.003333] Calibrating delay using timer specific routine.. 5602.44 > BogoMIPS (lpj=9333513) > > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > > [ 0.003333] CPU 2/2 -> Node 1 > > [ 0.003333] CPU: Physical Processor ID: 1 > > [ 0.003333] CPU: Processor Core ID: 0 > > [ 0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new > 0x7010600070106 > > [ 0.340530] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > > [ 0.348138] lockdep: fixing up alternatives. > > [ 0.349997] Booting processor 3/3 ip 6000 > > [ 0.003333] Initializing CPU#3 > > [ 0.003333] Calibrating delay using timer specific routine.. 5602.48 > BogoMIPS (lpj=9333575) > > [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 > bytes/line) > > [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) > > [ 0.003333] CPU 3/3 -> Node 1 > > [ 0.003333] CPU: Physical Processor ID: 1 > > [ 0.003333] CPU: Processor Core ID: 1 > > [ 0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new > 0x7010600070106 > > [ 0.447118] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 > > [ 0.454734] Brought up 4 CPUs > > [ 0.456640] Total of 4 processors activated (22409.64 BogoMIPS). > > [ 0.460033] Testing NMI watchdog ... > > [ 0.532432] WARNING: CPU#0: NMI appears to be stuck (0->0)! > > [ 0.533301] Please report this to bugzilla.kernel.org, > > [ 0.536635] and attach the output of the 'dmesg' command. > > [ 0.539968] > > [ 0.543302] WARNING: CPU#1: NMI appears to be stuck (0->0)! > > [ 0.546635] Please report this to bugzilla.kernel.org, > > [ 0.549968] and attach the output of the 'dmesg' command. > > [ 0.553301] > > [ 0.554980] WARNING: CPU#2: NMI appears to be stuck (0->0)! > > [ 0.556635] Please report this to bugzilla.kernel.org, > > [ 0.559968] and attach the output of the 'dmesg' command. > > [ 0.563301] > > [ 0.566635] WARNING: CPU#3: NMI appears to be stuck (0->0)! > > [ 0.569968] Please report this to bugzilla.kernel.org, > > [ 0.573301] and attach the output of the 'dmesg' command. > > [ 0.577183] net_namespace: 968 bytes > > [ 0.580160] xor: automatically using best checksumming function: > generic_sse > > [ 0.599969] generic_sse: 8630.400 MB/sec > > [ 0.603302] xor: using function: generic_sse (8630.400 MB/sec) > > [ 0.606692] NET: Registered protocol family 16 > > [ 0.610254] No dock devices found. > > [ 0.613503] node 0 link 2: io port [1000, 3fff] > > [ 0.613506] TOM: 00000000d0000000 aka 3328M > > [ 0.616636] node 0 link 2: mmio [d0000000, dfffffff] > > [ 0.616638] node 0 link 2: mmio [a0000, bffff] > > [ 0.616640] node 0 link 2: mmio [e0000000, e7ffffff] > > [ 0.616643] TOM2: 0000000230000000 aka 8960M > > [ 0.619969] bus: [00,ff] on node 0 link 2 > > [ 0.619970] bus: 00 index 0 io port: [0, ffff] > > [ 0.619972] bus: 00 index 1 mmio: [d0000000, ffffffff] > > [ 0.619974] bus: 00 index 2 mmio: [a0000, bffff] > > [ 0.619975] bus: 00 index 3 mmio: [230000000, fcffffffff] > > [ 0.619989] ACPI: bus type pci registered > > [ 0.623448] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - > 4 > > [ 0.626636] PCI: MCFG area at e0000000 reserved in E820 > > [ 0.630065] PCI: Using MMCONFIG at e0000000 - e04fffff > > [ 0.633302] PCI: Using configuration type 1 for base access > > [ 0.643980] ACPI: EC: Look up EC in DSDT > > [ 0.648409] ACPI: Interpreter enabled > > [ 0.649969] ACPI: (supports S0 S1 S3 S4 S5) > > [ 0.656635] ACPI: Using IOAPIC for interrupt routing > > [ 0.670034] ACPI: PCI Root Bridge [PCI0] (0000:00) > > [ 0.673616] pci 0000:00:01.1: PME# supported from D3hot D3cold > > [ 0.676644] pci 0000:00:01.1: PME# disabled > > [ 0.680009] pci 0000:00:02.0: supports D1 > > [ 0.680010] pci 0000:00:02.0: supports D2 > > [ 0.680012] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.683303] pci 0000:00:02.0: PME# disabled > > [ 0.686672] pci 0000:00:02.1: supports D1 > > [ 0.686673] pci 0000:00:02.1: supports D2 > > [ 0.686675] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.689969] pci 0000:00:02.1: PME# disabled > > [ 0.693535] pci 0000:00:08.0: supports D1 > > [ 0.693537] pci 0000:00:08.0: supports D2 > > [ 0.693538] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.696637] pci 0000:00:08.0: PME# disabled > > [ 0.700017] pci 0000:00:09.0: supports D1 > > [ 0.700019] pci 0000:00:09.0: supports D2 > > [ 0.700020] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.703304] pci 0000:00:09.0: PME# disabled > > [ 0.706664] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.709969] pci 0000:00:0a.0: PME# disabled > > [ 0.713331] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.716636] pci 0000:00:0d.0: PME# disabled > > [ 0.719997] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold > > [ 0.723302] pci 0000:00:0f.0: PME# disabled > > [ 0.726878] pci 0000:01:04.0: supports D1 > > [ 0.726879] pci 0000:01:04.0: supports D2 > > [ 0.726904] pci 0000:00:06.0: transparent bridge > > [ 0.729969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff] > > [ 0.733303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff] > > [ 0.736636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, > dfffffff] > > [ 0.740034] pci 0000:02:00.0: supports D1 > > [ 0.740064] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff] > > [ 0.743388] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] > > [ 0.743462] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT] > > [ 0.743486] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT] > > [ 0.743513] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT] > > [ 0.743540] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT] > > [ 0.769795] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.777165] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.786790] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.797164] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.808626] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.816220] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.822660] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.830122] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.840505] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.851950] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.858235] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.867922] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.877614] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.887303] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.898166] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 > 18 19 20 21 22 23) *0, disabled. > > [ 0.909241] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.915682] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.921892] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 > 18 19 20 21 22 23) > > [ 0.928367] Linux Plug and Play Support v0.97 (c) Adam Belay > > [ 0.930030] pnp: PnP ACPI init > > [ 0.933313] ACPI: bus type pnp registered > > [ 0.942967] pnp: PnP ACPI: found 15 devices > > [ 0.943302] ACPI: ACPI bus type pnp unregistered > > [ 0.946899] SCSI subsystem initialized > > [ 0.950132] libata version 3.00 loaded. > > [ 0.953483] usbcore: registered new interface driver usbfs > > [ 0.956699] usbcore: registered new interface driver hub > > [ 0.960060] usbcore: registered new device driver usb > > [ 0.963671] PCI: Using ACPI for IRQ routing > > [ 0.980227] PCI-DMA: Disabling AGP. > > [ 0.984431] PCI-DMA: aperture base @ 20000000 size 65536 KB > > [ 0.986635] PCI-DMA: using GART IOMMU. > > [ 0.989971] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture > > [ 0.993870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 > > [ 1.000341] hpet0: 3 32-bit timers, 25000000 Hz > > [ 1.004395] ACPI: RTC can wake from S4 > > [ 1.006637] Clockevents: could not switch to one-shot > mode:<6>Clockevents: could not switch to one-shot mode: lapic is not > functional. > > [ 1.009524] Could not switch to high resolution mode on CPU 3 > > [ 1.009528] Clockevents: could not switch to one-shot mode: lapic is not > functional. > > [ 1.009531] Could not switch to high resolution mode on CPU 1 > > [ 1.009534] Clockevents: could not switch to one-shot mode: lapic is not > functional. > > [ 1.009538] Could not switch to high resolution mode on CPU 2 > > [ 1.009969] lapic is not functional. > > [ 1.056944] Could not switch to high resolution mode on CPU 0 > > [ 1.076221] system 00:00: iomem range 0xffc00000-0xffffffff could not be > reserved > > [ 1.084054] system 00:00: iomem range 0xfec00000-0xfec00fff could not be > reserved > > [ 1.091883] system 00:00: iomem range 0xfee00000-0xfeefffff could not be > reserved > > [ 1.099719] system 00:00: iomem range 0xfed00000-0xfed00fff has been > reserved > > [ 1.107052] system 00:03: iomem range 0xd0000000-0xd0007fff has been > reserved > > [ 1.114383] system 00:04: ioport range 0x1000-0x107f has been reserved > > [ 1.121101] system 00:04: ioport range 0x1080-0x10ff has been reserved > > [ 1.127817] system 00:04: ioport range 0x1400-0x147f has been reserved > > [ 1.134537] system 00:04: ioport range 0x1480-0x14ff has been reserved > > [ 1.141258] system 00:04: ioport range 0x1800-0x187f has been reserved > > [ 1.147976] system 00:04: ioport range 0x1880-0x18ff has been reserved > > [ 1.154698] system 00:04: ioport range 0x2440-0x247f has been reserved > > [ 1.161420] system 00:04: ioport range 0x2400-0x243f has been reserved > > [ 1.168151] system 00:07: ioport range 0x4d0-0x4d1 has been reserved > > [ 1.174695] system 00:07: ioport range 0xc05-0xc06 has been reserved > > [ 1.182277] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01 > > [ 1.188388] pci 0000:00:06.0: IO window: 0x3000-0x3fff > > [ 1.193889] pci 0000:00:06.0: MEM window: 0xd0100000-0xd01fffff > > [ 1.200176] pci 0000:00:06.0: PREFETCH window: > 0x000000d8000000-0x000000dfffffff > > [ 1.208089] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02 > > [ 1.214205] pci 0000:00:0a.0: IO window: disabled > > [ 1.219275] pci 0000:00:0a.0: MEM window: 0xd0200000-0xd02fffff > > [ 1.225558] pci 0000:00:0a.0: PREFETCH window: > 0x000000d1000000-0x000000d10fffff > > [ 1.233474] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03 > > [ 1.239598] pci 0000:00:0d.0: IO window: disabled > > [ 1.244666] pci 0000:00:0d.0: MEM window: disabled > > [ 1.249825] pci 0000:00:0d.0: PREFETCH window: disabled > > [ 1.255416] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04 > > [ 1.261525] pci 0000:00:0f.0: IO window: disabled > > [ 1.266594] pci 0000:00:0f.0: MEM window: disabled > > [ 1.271751] pci 0000:00:0f.0: PREFETCH window: disabled > > [ 1.277347] pci 0000:00:06.0: setting latency timer to 64 > > [ 1.277352] pci 0000:00:0a.0: setting latency timer to 64 > > [ 1.277357] pci 0000:00:0d.0: setting latency timer to 64 > > [ 1.277363] pci 0000:00:0f.0: setting latency timer to 64 > > [ 1.277365] bus: 00 index 0 io port: [0, ffff] > > [ 1.282004] bus: 00 index 1 mmio: [0, ffffffffffffffff] > > [ 1.292287] bus: 01 index 0 io port: [3000, 3fff] > > [ 1.297187] bus: 01 index 1 mmio: [d0100000, d01fffff] > > [ 1.302519] bus: 01 index 2 mmio: [d8000000, dfffffff] > > [ 1.307849] bus: 01 index 3 io port: [0, ffff] > > [ 1.312483] bus: 01 index 4 mmio: [0, ffffffffffffffff] > > [ 1.317900] bus: 02 index 0 mmio: [0, 0] > > [ 1.322017] bus: 02 index 1 mmio: [d0200000, d02fffff] > > [ 1.327349] bus: 02 index 2 mmio: [d1000000, d10fffff] > > [ 1.332678] bus: 02 index 3 mmio: [0, 0] > > [ 1.336793] bus: 03 index 0 mmio: [0, 0] > > [ 1.340910] bus: 03 index 1 mmio: [0, 0] > > [ 1.345025] bus: 03 index 2 mmio: [0, 0] > > [ 1.349142] bus: 03 index 3 mmio: [0, 0] > > [ 1.353256] bus: 04 index 0 mmio: [0, 0] > > [ 1.357374] bus: 04 index 1 mmio: [0, 0] > > [ 1.361490] bus: 04 index 2 mmio: [0, 0] > > [ 1.365609] bus: 04 index 3 mmio: [0, 0] > > [ 1.369739] NET: Registered protocol family 2 > > [ 1.413137] IP route cache hash table entries: 262144 (order: 9, 2097152 > bytes) > > [ 1.423160] TCP established hash table entries: 524288 (order: 11, > 8388608 bytes) > > [ 1.436206] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes) > > [ 1.445874] TCP: Hash tables configured (established 524288 bind 65536) > > [ 1.452716] TCP reno registered > > [ 1.466350] NET: Registered protocol family 1 > > [ 1.471023] checking if image is initramfs... it is > > [ 1.572646] Freeing initrd memory: 1921k freed > > [ 1.578336] Simple Boot Flag at 0x37 set to 0x80 > > [ 1.585905] audit: initializing netlink socket (disabled) > > [ 1.591560] type=2000 audit(1221176947.589:1): initialized > > [ 1.602577] HugeTLB registered 2 MB page size, pre-allocated 0 pages > > [ 1.613210] VFS: Disk quotas dquot_6.5.1 > > [ 1.617471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) > > [ 1.625780] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). > > [ 1.632809] msgmni has been set to 15840 > > [ 1.637191] async_tx: api initialized (async) > > [ 1.641747] io scheduler noop registered > > [ 1.645871] io scheduler anticipatory registered > > [ 1.650694] io scheduler deadline registered > > [ 1.655347] io scheduler cfq registered (default) > > [ 1.660281] pci 0000:00:00.0: Enabling HT MSI Mapping > > [ 1.879582] pci 0000:00:05.0: Enabling HT MSI Mapping > > [ 1.884851] pci 0000:00:05.1: Enabling HT MSI Mapping > > [ 1.890123] pci 0000:00:05.2: Enabling HT MSI Mapping > > [ 1.895393] pci 0000:00:06.0: Enabling HT MSI Mapping > > [ 1.900669] pci 0000:00:08.0: Enabling HT MSI Mapping > > [ 1.905945] pci 0000:00:09.0: Enabling HT MSI Mapping > > [ 1.911213] pci 0000:00:0a.0: Enabling HT MSI Mapping > > [ 1.916486] pci 0000:00:0d.0: Enabling HT MSI Mapping > > [ 1.921761] pci 0000:00:0f.0: Enabling HT MSI Mapping > > [ 1.927029] pci 0000:01:04.0: Boot video device > > [ 1.927212] pcieport-driver 0000:00:0a.0: setting latency timer to 64 > > [ 1.927241] pcieport-driver 0000:00:0a.0: found MSI capability > > [ 1.933324] pci_express 0000:00:0a.0:pcie00: allocate port service > > [ 1.933414] pci_express 0000:00:0a.0:pcie01: allocate port service > > [ 1.933475] pci_express 0000:00:0a.0:pcie03: allocate port service > > [ 1.933576] pcieport-driver 0000:00:0d.0: setting latency timer to 64 > > [ 1.933604] pcieport-driver 0000:00:0d.0: found MSI capability > > [ 1.939667] pci_express 0000:00:0d.0:pcie00: allocate port service > > [ 1.939741] pci_express 0000:00:0d.0:pcie01: allocate port service > > [ 1.939801] pci_express 0000:00:0d.0:pcie03: allocate port service > > [ 1.939898] pcieport-driver 0000:00:0f.0: setting latency timer to 64 > > [ 1.939925] pcieport-driver 0000:00:0f.0: found MSI capability > > [ 1.945977] pci_express 0000:00:0f.0:pcie00: allocate port service > > [ 1.946048] pci_express 0000:00:0f.0:pcie01: allocate port service > > [ 1.946108] pci_express 0000:00:0f.0:pcie03: allocate port service > > [ 1.947934] aer 0000:00:0a.0:pcie01: AER service couldn't init device: > no _OSC support > > [ 1.949554] aer 0000:00:0d.0:pcie01: AER service couldn't init device: > no _OSC support > > [ 1.951148] aer 0000:00:0f.0:pcie01: AER service couldn't init device: > no _OSC support > > [ 1.951606] input: Power Button (FF) as /class/input/input0 > > [ 1.957384] ACPI: Power Button (FF) [PWRF] > > [ 1.961801] input: Power Button (CM) as /class/input/input1 > > [ 1.967565] ACPI: Power Button (CM) [PWRB] > > [ 1.972294] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) > > [ 1.978363] processor ACPI0007:00: registered as cooling_device0 > > [ 1.984576] ACPI: Processor [C000] (supports 2 throttling states) > > [ 1.991063] processor ACPI0007:01: registered as cooling_device1 > > [ 1.997366] processor ACPI0007:02: registered as cooling_device2 > > [ 2.003684] processor ACPI0007:03: registered as cooling_device3 > > [ 2.079999] hpet_resources: 0xfed00000 is busy > > [ 2.080193] Non-volatile memory driver v1.2 > > [ 2.084715] Linux agpgart interface v0.103 > > [ 2.089107] Serial: 8250/16550 driver4 ports, IRQ sharing disabled > > [ 2.095680] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A > > [ 2.102659] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A > > [ 2.108789] Floppy drive(s): fd0 is 1.44M > > [ 2.130069] FDC 0 is a post-1991 82077 > > [ 2.137592] brd: module loaded > > [ 2.142041] loop: module loaded > > [ 2.145379] tun: Universal TUN/TAP device driver, 1.6 > > [ 2.150619] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> > > [ 2.157181] console [netcon0] enabled > > [ 2.161035] netconsole: network logging started > > [ 2.165762] Uniform Multi-Platform E-IDE driver > > [ 2.170615] ide_generic: please use "probe_mask=0x3f" module parameter > for probing all legacy ISA IDE ports > > [ 2.180715] Probing IDE interface ide0... > > [ 3.149712] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive > > [ 3.206282] Probing IDE interface ide1... > > [ 3.739617] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 > > [ 3.744443] ide1 at 0x170-0x177,0x376 on irq 15 > > [ 3.751258] hdb: ATAPI 24X CD-ROM drive, 256kB Cache > > [ 3.756646] Uniform CD-ROM driver Revision: 3.20 > > [ 3.771462] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 > EST 2006) > > [ 3.779263] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST > 2006) > > [ 3.786512] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008 > > [ 3.792690] Driver 'sd' needs updating - please use bus_type methods > > [ 3.799315] Driver 'sr' needs updating - please use bus_type methods > > [ 3.806449] sata_nv 0000:00:05.0: version 3.5 > > [ 3.806741] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23 > > [ 3.812700] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 > (level, low) -> IRQ 23 > > [ 3.821311] sata_nv 0000:00:05.0: Using SWNCQ mode > > [ 3.826430] sata_nv 0000:00:05.0: setting latency timer to 64 > > [ 3.826613] scsi0 : sata_nv > > [ 3.829796] scsi1 : sata_nv > > [ 3.832986] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 > irq 23 > > [ 3.840314] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 > irq 23 > > [ 4.179871] ata1: SATA link down (SStatus 0 SControl 300) > > [ 4.516536] ata2: SATA link down (SStatus 0 SControl 300) > > [ 4.522429] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22 > > [ 4.528388] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 > (level, low) -> IRQ 22 > > [ 4.536997] sata_nv 0000:00:05.1: Using SWNCQ mode > > [ 4.542083] sata_nv 0000:00:05.1: setting latency timer to 64 > > [ 4.542220] scsi2 : sata_nv > > [ 4.545334] scsi3 : sata_nv > > [ 4.548512] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 > irq 22 > > [ 4.555840] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 > irq 22 > > [ 4.896535] ata3: SATA link down (SStatus 0 SControl 300) > > [ 5.233200] ata4: SATA link down (SStatus 0 SControl 300) > > [ 5.239087] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21 > > [ 5.245047] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 > (level, low) -> IRQ 21 > > [ 5.253654] sata_nv 0000:00:05.2: Using SWNCQ mode > > [ 5.258744] sata_nv 0000:00:05.2: setting latency timer to 64 > > [ 5.258881] scsi4 : sata_nv > > [ 5.261998] scsi5 : sata_nv > > [ 5.265174] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 > irq 21 > > [ 5.272504] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 > irq 21 > > [ 5.613197] ata5: SATA link down (SStatus 0 SControl 300) > > [ 5.949864] ata6: SATA link down (SStatus 0 SControl 300) > > [ 5.955612] pata_amd 0000:00:04.0: version 0.3.10 > > [ 5.955640] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region > [0x1f0-0x1f7] > > [ 5.963385] pata_amd 0000:00:04.0: failed to request/iomap BARs for port > 0 (errno=-16) > > [ 5.971653] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region > [0x170-0x177] > > [ 5.979396] pata_amd 0000:00:04.0: failed to request/iomap BARs for port > 1 (errno=-16) > > [ 5.987656] pata_amd 0000:00:04.0: no available native port > > [ 5.993731] Fusion MPT base driver 3.04.07 > > [ 5.998021] Copyright (c) 1999-2008 LSI Corporation > > [ 6.003109] Fusion MPT SPI Host driver 3.04.07 > > [ 6.007833] Fusion MPT SAS Host driver 3.04.07 > > [ 6.013164] ieee1394: raw1394: /dev/raw1394 device initialized > > [ 6.019695] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20 > > [ 6.025650] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 > (level, low) -> IRQ 20 > > [ 6.034353] ehci_hcd 0000:00:02.1: setting latency timer to 64 > > [ 6.034355] ehci_hcd 0000:00:02.1: EHCI Host Controller > > [ 6.040002] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus > number 1 > > [ 6.047776] ehci_hcd 0000:00:02.1: debug port 1 > > [ 6.052503] ehci_hcd 0000:00:02.1: cache line size of 64 is not > supported > > [ 6.052521] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000 > > [ 6.066176] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 > Dec 2004 > > [ 6.074154] usb usb1: configuration #1 chosen from 1 choice > > [ 6.080016] hub 1-0:1.0: USB hub found > > [ 6.083976] hub 1-0:1.0: 10 ports detected > > [ 6.189886] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller > (OHCI) Driver > > [ 6.190142] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19 > > [ 6.196100] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 > (level, low) -> IRQ 19 > > [ 6.204801] ohci_hcd 0000:00:02.0: setting latency timer to 64 > > [ 6.204804] ohci_hcd 0000:00:02.0: OHCI Host Controller > > [ 6.210376] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus > number 2 > > [ 6.218132] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000 > > [ 6.278315] usb usb2: configuration #1 chosen from 1 choice > > [ 6.284165] hub 2-0:1.0: USB hub found > > [ 6.288118] hub 2-0:1.0: 10 ports detected > > [ 6.393142] USB Universal Host Controller Interface driver v3.0 > > [ 6.399423] usbcore: registered new interface driver usblp > > [ 6.405107] Initializing USB Mass Storage driver... > > [ 6.410243] usbcore: registered new interface driver usb-storage > > [ 6.416443] USB Mass Storage support registered. > > [ 6.421406] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at > 0x60,0x64 irq 1,12 > > [ 6.679856] serio: i8042 KBD port at 0x60,0x64 irq 1 > > [ 6.685238] mice: PS/2 mouse device common for all mice > > [ 6.877983] md: linear personality registered for level -1 > > [ 6.883663] md: raid0 personality registered for level 0 > > [ 6.889166] md: raid1 personality registered for level 1 > > [ 6.894669] md: raid10 personality registered for level 10 > > [ 6.956180] raid6: int64x1 2587 MB/s > > [ 7.016180] raid6: int64x2 3291 MB/s > > [ 7.076168] raid6: int64x4 3160 MB/s > > [ 7.136168] raid6: int64x8 2360 MB/s > > [ 7.196178] raid6: sse2x1 3858 MB/s > > [ 7.256175] raid6: sse2x2 5139 MB/s > > [ 7.316176] raid6: sse2x4 5262 MB/s > > [ 7.320122] raid6: using algorithm sse2x4 (5262 MB/s) > > [ 7.325365] md: raid6 personality registered for level 6 > > [ 7.335744] md: raid5 personality registered for level 5 > > [ 7.341252] md: raid4 personality registered for level 4 > > [ 7.346753] md: multipath personality registered for level -4 > > [ 7.352960] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: > dm-devel@redhat.com > > [ 7.361877] cpuidle: using governor ladder > > [ 7.366402] cpuidle: using governor menu > > [ 7.370591] usbcore: registered new interface driver usbhid > > [ 7.376362] usbhid: v2.6:USB HID core driver > > [ 7.380936] oprofile: using NMI interrupt. > > [ 7.385695] TCP cubic registered > > [ 7.389332] NET: Registered protocol family 10 > > [ 7.394985] IPv6 over IPv4 tunneling driver > > [ 7.400135] NET: Registered protocol family 17 > > [ 7.405122] RPC: Registered udp transport module. > > [ 7.410021] RPC: Registered tcp transport module. > > [ 7.415002] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor > 2220 processors (4 cpu cores) (version 2.20.00) > > [ 7.425975] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 > > [ 7.431831] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa > > [ 7.437687] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc > > [ 7.443537] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe > > [ 7.449301] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 > > [ 7.455149] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 > > [ 7.460999] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 > > [ 7.467050] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 > > [ 7.472929] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa > > [ 7.478785] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc > > [ 7.484637] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe > > [ 7.490411] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 > > [ 7.496279] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 > > [ 7.502134] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 > > [ 7.508739] drivers/rtc/hctosys.c: unable to open rtc device (rtc0) > > [ 7.515289] Freeing unused kernel memory: 676k freed > > [ 7.520818] Write protecting the kernel read-only data: 5672k > > [ 7.742843] Clocksource tsc unstable (delta = -111149082 ns) > > [ 8.177545] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18 > > [ 8.177654] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 > (level, low) -> IRQ 18 > > [ 8.177666] arcmsr 0000:02:00.0: setting latency timer to 64 > > [ 8.189529] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31 > > [ 8.202850] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 > capable) > > [ 8.202853] Driver Version 1.20.00.15 2008/02/27 > > [ 8.205018] scsi 6:0:0:0: Direct-Access Areca ARC-1280-VOL#00 > R001 PQ: 0 ANSI: 5 > > [ 8.207052] sd 6:0:0:0: [sda] Very big device. Trying to use READ > CAPACITY(16). > > [ 8.207117] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors > (10999996 MB) > > [ 8.207182] sd 6:0:0:0: [sda] Write Protect is off > > [ 8.207188] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08 > > [ 8.207298] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, > doesn't support DPO or FUA > > [ 8.207560] sd 6:0:0:0: [sda] Very big device. Trying to use READ > CAPACITY(16). > > [ 8.207621] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors > (10999996 MB) > > [ 8.207685] sd 6:0:0:0: [sda] Write Protect is off > > [ 8.207692] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08 > > [ 8.207802] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, > doesn't support DPO or FUA > > [ 8.207811] sda: sda1 sda2 sda3 sda4 > > [ 8.245400] sd 6:0:0:0: [sda] Attached SCSI disk > > [ 8.247390] sd 6:0:0:0: Attached scsi generic sg0 type 0 > > [ 8.249720] scsi 6:0:16:0: Processor Areca RAID controller > R001 PQ: 0 ANSI: 0 > > [ 8.252259] scsi 6:0:16:0: Attached scsi generic sg1 type 3 > > [ 8.504889] 3ware Storage Controller device driver for Linux > v1.26.02.002. > > [ 8.542592] 3ware 9000 Storage Controller device driver for Linux > v2.26.02.011. > > [ 8.601009] Adaptec aacraid driver 1.1-5[2456]-ms > > [ 8.993029] SGI XFS with ACLs, security attributes, large block/inode > numbers, no debug enabled > > [ 8.998828] SGI XFS Quota Management subsystem > > [ 9.048982] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI > > [ 9.048990] Copyright (c) 1999-2006 Intel Corporation. > > [ 12.346739] EXT3-fs: INFO: recovery required on readonly filesystem. > > [ 12.346748] EXT3-fs: write access will be enabled during recovery. > > [ 13.161202] kjournald starting. Commit interval 5 seconds > > [ 13.161240] EXT3-fs: sda3: orphan cleanup on readonly fs > > [ 13.161256] ext3_orphan_cleanup: deleting unreferenced inode 2501876 > > [ 13.161508] ext3_orphan_cleanup: deleting unreferenced inode 2501875 > > [ 13.161529] ext3_orphan_cleanup: deleting unreferenced inode 2501874 > > [ 13.161549] ext3_orphan_cleanup: deleting unreferenced inode 2501873 > > [ 13.161565] EXT3-fs: sda3: 4 orphan inodes deleted > > [ 13.161570] EXT3-fs: recovery complete. > > [ 13.162085] EXT3-fs: mounted filesystem with ordered data mode. > > [ 15.634222] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440 > > [ 15.634257] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400 > > [ 15.660203] forcedeth: Reverse Engineered nForce ethernet driver. > Version 0.61. > > [ 15.660582] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17 > > [ 15.660659] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 > (level, low) -> IRQ 17 > > [ 15.660663] forcedeth 0000:00:08.0: setting latency timer to 64 > > [ 15.663568] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, > addr 00:e0:81:76:8b:be > > [ 15.663571] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq > gbit lnktim msi desc-v3 > > [ 15.663876] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16 > > [ 15.663898] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 > (level, low) -> IRQ 16 > > [ 15.663923] forcedeth 0000:00:09.0: setting latency timer to 64 > > [ 15.665255] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, > addr 00:e0:81:76:8b:bf > > [ 15.665257] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq > gbit lnktim msi desc-v3 > > [ 18.107517] EXT3 FS on sda3, internal journal > > [ 18.380032] SMsC 37B787 watchdog component driver 1.1 initialising... > > [ 18.381424] smsc37b787_wdt: Timeout set to 60 second(s). > > [ 18.381429] smsc37b787_wdt: Watchdog initialized and sleeping > (nowayout=0)... > > [ 21.417259] kjournald starting. Commit interval 5 seconds > > [ 21.417297] EXT3-fs warning: checktime reached, running e2fsck is > recommended > > [ 21.422349] EXT3 FS on sda4, internal journal > > [ 21.422359] EXT3-fs: recovery complete. > > [ 21.422430] EXT3-fs: mounted filesystem with ordered data mode. > > [ 21.501039] Adding 15999992k swap on /dev/sda2. Priority:-1 extents:1 > across:15999992k > > [ 42.038582] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state > recovery directory > > [ 42.045903] NFSD: starting 90-second grace period > > [ 45.396027] eth0: no IPv6 routers present > > [ 49.761642] w83627hf: Found W83627HF chip at 0xc00 > On Mon, 15 Sep 2008, Joshua Hoblitt wrote: > In addition to the deadlocks, we still have the watchdog warning: > > [ 0.460034] Testing NMI watchdog ... > [ 0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)! > [ 0.533301] Please report this to bugzilla.kernel.org, > [ 0.536635] and attach the output of the 'dmesg' command. > > Perhaps an HPET problem: > > [ 0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 > [ 0.999969] hpet0: 3 32-bit timers, 25000000 Hz > [ 1.004396] ACPI: RTC can wake from S4 > [ 1.006637] Clockevents: could not switch to one-shot > mode:<6>Clockevents: could not switch to one-shot mode: lapic is not > functional. > [ 1.009844] Could not switch to high resolution mode on CPU 3 > [ 1.009848] Clockevents: could not switch to one-shot mode: lapic is not > functional. > [ 1.009852] Could not switch to high resolution mode on CPU 2 > [ 1.009855] Clockevents: could not switch to one-shot mode: lapic is not > functional. > [ 1.009858] Could not switch to high resolution mode on CPU 1 > [ 1.009969] lapic is not functional. > [ 1.056944] Could not switch to high resolution mode on CPU 0 No, that's documented behaviour: > > > [ 0.126660] APIC timer registered as dummy, due to nmi_watchdog=1! Can you try nmi_watchdog=2 ? Thanks, tglx [Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700] | On Mon, 15 Sep 2008, Joshua Hoblitt wrote: | | > In addition to the deadlocks, we still have the watchdog warning: | > | > [ 0.460034] Testing NMI watchdog ... | > [ 0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)! | > [ 0.533301] Please report this to bugzilla.kernel.org, | > [ 0.536635] and attach the output of the 'dmesg' command. | > | > Perhaps an HPET problem: | > | > [ 0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 | > [ 0.999969] hpet0: 3 32-bit timers, 25000000 Hz | > [ 1.004396] ACPI: RTC can wake from S4 | > [ 1.006637] Clockevents: could not switch to one-shot | > mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional. | > [ 1.009844] Could not switch to high resolution mode on CPU 3 | > [ 1.009848] Clockevents: could not switch to one-shot mode: lapic is not functional. | > [ 1.009852] Could not switch to high resolution mode on CPU 2 | > [ 1.009855] Clockevents: could not switch to one-shot mode: lapic is not functional. | > [ 1.009858] Could not switch to high resolution mode on CPU 1 | > [ 1.009969] lapic is not functional. | > [ 1.056944] Could not switch to high resolution mode on CPU 0 | | No, that's documented behaviour: | | > > > [ 0.126660] APIC timer registered as dummy, due to nmi_watchdog=1! | | Can you try nmi_watchdog=2 ? | | Thanks, | | tglx | And get apic=debug a try too please. I remember there was a problem with SB600 on ACPI side (but they should be already fixed) - Cyrill - [Cyrill Gorcunov - Tue, Sep 16, 2008 at 09:56:29PM +0400] ... | | And get apic=debug a try too please. I remember there | was a problem with SB600 on ACPI side (but they should | be already fixed) | | - Cyrill - Sorry Thomas, I meant to send the message to Joshua - Cyrill - In-Reply-To: <20080916175629.GE7187@lenovo>
On Tue, Sep 16, 2008 at 09:56:29PM +0400, Cyrill Gorcunov wrote:
> [Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700]
> | Can you try nmi_watchdog=2 ?
> |
> | Thanks,
> |
> | tglx
> |
>
> And get apic=debug a try too please. I remember there
> was a problem with SB600 on ACPI side (but they should
> be already fixed)
Attached is the updated dmesg from adding the nmi_watchdog=2 and acpi=debug boot params.
-J
--
[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 04:43:12PM -1000] | In-Reply-To: <20080916175629.GE7187@lenovo> | On Tue, Sep 16, 2008 at 09:56:29PM +0400, Cyrill Gorcunov wrote: | > [Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700] | > | Can you try nmi_watchdog=2 ? | > | | > | Thanks, | > | | > | tglx | > | | > | > And get apic=debug a try too please. I remember there | > was a problem with SB600 on ACPI side (but they should | > be already fixed) | | Attached is the updated dmesg from adding the nmi_watchdog=2 and acpi=debug boot params. | | -J | Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)? Not sure if it help but could be usefull info. Thanks for this dmesg - trying to analize it. ( here is how to do it http://kernel.org/pub/linux/kernel/people/helgaas/debug http://lwn.net/Articles/237085/ ) - Cyrill - Created attachment 17832 [details]
acpi table dump
On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote:
> Joshua, could you please attach ACPI tables dump to the bugzilla entry
> (~300K)?
Done but I was unable to extract the DSDT table as described in the
instructions.
# acpixtract DSDT < acpidump.asc
Could not open DSDT
Cheers,
-J
--
[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000] | On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote: | > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)? | | Done but I was unable to extract the DSDT table as described in the | instructions. | | # acpixtract DSDT < acpidump.asc | Could not open DSDT | | Cheers, | | -J | it's enough, thanks! - Cyrill - [j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000] | On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote: | > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)? | | Done but I was unable to extract the DSDT table as described in the | instructions. | | # acpixtract DSDT < acpidump.asc | Could not open DSDT | | Cheers, | | -J | | -- | As far as I can see there is no really issue in ACPI configuration, at least on apic side... Evetually nmi_watchdog=2 does work right? Though it's not good that we've got 8259 spurious interrupt. Hmm... - Cyrill -
> Evetually nmi_watchdog=2 does work right? Though it's not good that
> we've got 8259 spurious interrupt. Hmm...
well, nmi_watchdog=2 changes the layout of clockevent devices and easily
switches the system into non-highres non-dynticks mode. So it can hide
bugs.
Ingo
[Ingo Molnar - Wed, Sep 17, 2008 at 03:13:06PM +0200] | | > Evetually nmi_watchdog=2 does work right? Though it's not good that | > we've got 8259 spurious interrupt. Hmm... | | well, nmi_watchdog=2 changes the layout of clockevent devices and easily | switches the system into non-highres non-dynticks mode. So it can hide | bugs. | | Ingo | and it does since we have nmi stuck on lapics. Still investigating. - Cyrill - [j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000] | On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote: | > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)? | | Done but I was unable to extract the DSDT table as described in the | instructions. | | # acpixtract DSDT < acpidump.asc | Could not open DSDT | | Cheers, | | -J | | -- | Joshua, could you please do one more thing - boot the kernel with "debug apic=debug nmi_watchdog=1" (among others) and publish dmsg? - Cyrill - > > Evetually nmi_watchdog=2 does work right? Though it's not good that
> > we've got 8259 spurious interrupt. Hmm...
>
> well, nmi_watchdog=2 changes the layout of clockevent devices and easily
> switches the system into non-highres non-dynticks mode. So it can hide
> bugs.
No, nmi_watchdog=1 is the one which goes into non-highres non-dynticks
mode. That's why I asked to use nmi_watchdog=2
Thanks,
tglx
On Wed, Sep 17, 2008 at 08:27:04PM +0400, Cyrill Gorcunov wrote:
> Joshua, could you please do one more thing - boot the kernel
> with "debug apic=debug nmi_watchdog=1" (among others) and
> publish dmsg?
dmesg from "debug apic=debug nmi_watchdog=2" is attached.
^
-J
--
[j_kernel@hoblitt.com - Wed, Sep 17, 2008 at 10:48:50AM -1000] | On Wed, Sep 17, 2008 at 08:27:04PM +0400, Cyrill Gorcunov wrote: | > Joshua, could you please do one more thing - boot the kernel | > with "debug apic=debug nmi_watchdog=1" (among others) and | > publish dmsg? | | dmesg from "debug apic=debug nmi_watchdog=2" is attached. | ^ | | -J | | -- ... No Joshua, exactly 'nmi_watchdog=1' since it's important to find who is responsible that cpu is stuck while testing nmi watchdog. 'debug and apic=debug' will show additional info what is happening. And it would be just great if you able to try latest -tip/master http://people.redhat.com/mingo/tip.git/README since there a lot of work/fix are done. If it's a problem to fetch this kernel thru git - me or Ingo (I suppose) could make tar.bz2 archive so you'll not need to fetch the whole history. I wouldn't say I'm really specialist in this area so your report could be quite important not for me only but for others too. - Cyrill - References : http://marc.info/?l=linux-kernel&m=122117786124326&w=4 Handled-By : Thomas Gleixner <tglx@linutronix.de> Handled-By : Cyrill Gorcunov <gorcunov@gmail.com> Handled-By : Ingo Molnar <mingo@elte.hu> Same issue or is this a separate problem with the watchdog? ipp018 login: [ 717.820766] BUG: NMI Watchdog detected LOCKUP on CPU2, ip ffffffff80211f60, registers: [ 717.820766] CPU 2 [ 717.820766] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan [ 717.820766] Pid: 0, comm: swapper Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2 [ 717.820766] RIP: 0010:[<ffffffff80211f60>] [<ffffffff80211f60>] native_read_tsc+0xb/0x18 [ 717.820766] RSP: 0018:ffff88012fab7db0 EFLAGS: 00000002 [ 717.820766] RAX: 000000007d8e8ab6 RBX: 0000000000000001 RCX: 000000000004a007 [ 717.820766] RDX: 0000000000000205 RSI: 0000000000000000 RDI: 0000000000000001 [ 717.820766] RBP: 000000007d8e8a5e R08: 0000000000000001 R09: 0000000000000001 [ 717.820766] R10: 0000000000000000 R11: ffffffff803bac5e R12: 0000000000000002 [ 717.820766] R13: 00000000a7c04084 R14: 0000000000000800 R15: 0000000000000000 [ 717.820766] FS: 00000000006f5360(0000) GS:ffff88022fa22300(0000) knlGS:0000000000000000 [ 717.820766] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 717.820766] CR2: 00007fbf400750e0 CR3: 000000022dc37000 CR4: 00000000000006e0 [ 717.820766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 717.820766] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 717.820766] Process swapper (pid: 0, threadinfo ffff88022fa5e000, task ffff88012fa83c40) [ 717.820766] Stack: ffffffff803bac7a ffff880136257100 0000000004b06ebc 0000000000000001 [ 717.820766] ffffffff803be1b7 0000000000000000 ffff880136257100 ffff88012fab7e68 [ 717.820766] 0000000000000092 ffff880136257100 ffffffff805c7f70 ffffffff802319b9 [ 717.820766] Call Trace: [ 717.820766] <IRQ> [<ffffffff803bac7a>] ? delay_tsc+0x1c/0x42 [ 717.820766] [<ffffffff803be1b7>] ? _raw_spin_lock+0xb8/0x120 [ 717.820766] [<ffffffff805c7f70>] ? _spin_lock_irqsave+0x37/0x3f [ 717.820766] [<ffffffff802319b9>] ? tg_shares_up+0xcf/0x195 [ 717.820766] [<ffffffff802319b9>] ? tg_shares_up+0xcf/0x195 [ 717.820766] [<ffffffff802318ea>] ? tg_shares_up+0x0/0x195 [ 717.820766] [<ffffffff80229658>] ? tg_nop+0x0/0x6 [ 717.820766] [<ffffffff8022ad28>] ? walk_tg_tree+0x90/0xbc [ 717.820766] [<ffffffff8022ac98>] ? walk_tg_tree+0x0/0xbc [ 717.820766] [<ffffffff8022e2ed>] ? try_to_wake_up+0xb0/0x238 [ 717.820766] [<ffffffff80249fa8>] ? hrtimer_wakeup+0x0/0x21 [ 717.820766] [<ffffffff80249fc5>] ? hrtimer_wakeup+0x1d/0x21 [ 717.820766] [<ffffffff80249dba>] ? __run_hrtimer+0x55/0x92 [ 717.820766] [<ffffffff8024a7e7>] ? hrtimer_interrupt+0xe3/0x15d [ 717.820766] [<ffffffff8021c23c>] ? smp_apic_timer_interrupt+0x89/0xa9 [ 717.820766] [<ffffffff8020cbb6>] ? apic_timer_interrupt+0x66/0x70 [ 717.820766] <EOI> [<ffffffff805ca561>] ? __atomic_notifier_call_chain+0x0/0x83 [ 717.820766] [<ffffffff805ca561>] ? __atomic_notifier_call_chain+0x0/0x83 [ 717.820766] [<ffffffff80212280>] ? default_idle+0x27/0x3b [ 717.820766] [<ffffffff80212469>] ? c1e_idle+0xd4/0xd8 [ 717.820766] [<ffffffff8020abd1>] ? cpu_idle+0x7b/0xa1 [ 717.820766] [ 717.820766] [ 717.820766] Code: 8c 1a 00 c3 90 90 90 40 0f b6 c7 e6 70 e4 71 0f b6 c0 c3 40 0f b6 c6 e6 70 40 0f b6 c7 e6 71 c3 0f ae f0 66 66 90 0f 31 0f ae f0 <66> 66 90 48 c1 e2 20 89 c0 48 09 d0 c3 41 57 41 56 41 55 41 54 [ 717.820766] ---[ end trace 32605f607ce8ad50 ]--- [ 717.820766] Kernel panic - not syncing: Aiee, killing interrupt handler! [ 717.821183] BUG: NMI Watchdog detected LOCKUP on CPU3, ip ffffffff8024bbf4, registers: [ 717.821183] CPU 3 [ 717.821183] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan [ 717.821183] Pid: 0, comm: swapper Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2 [ 717.821183] RIP: 0010:[<ffffffff8024bbf4>] [<ffffffff8024bbf4>] sched_clock_cpu+0x116/0x12b [ 717.821183] RSP: 0018:ffff88022fa93e58 EFLAGS: 00000002 [ 717.821183] RAX: 000000000000292a RBX: ffffffff80919a80 RCX: ffffffff8090f478 [ 717.821183] RDX: ffff880136253040 RSI: 000000b62fead5f3 RDI: 000000b62fead5f3 [ 717.821183] RBP: ffff880136257a80 R08: 0000000000000001 R09: 0000000000010000 [ 717.821183] R10: ffff8801b593e000 R11: 0000000000000000 R12: 0000000000000003 [ 717.821183] R13: ffffffff80915310 R14: 0000000000000018 R15: 0000000000000003 [ 717.821183] FS: 00007fbf407036f0(0000) GS:ffff88022fa22780(0000) knlGS:0000000000000000 [ 717.821183] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 717.821183] CR2: 0000000000615198 CR3: 000000020e43a000 CR4: 00000000000006e0 [ 717.821183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 717.821183] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 717.821183] Process swapper (pid: 0, threadinfo ffff88012fadc000, task ffff88022fa88000) [ 717.821183] Stack: 0000000000000096 0000000000000000 ffffffff80915318 ffffffff8024bd13 [ 717.821183] 000000000000029c ffffffff8027233f 000000010001e9d8 ffffffff8027247a [ 717.821183] ffff88012fadde78 0000000000000003 0000000000000000 ffff88022fa88000 [ 717.821183] Call Trace: [ 717.821183] <IRQ> [<ffffffff8024bd13>] ? cpu_clock+0x9/0xe [ 717.821183] [<ffffffff8027233f>] ? get_timestamp+0x9/0xf [ 717.821183] [<ffffffff8027247a>] ? softlockup_tick+0xc7/0x1ab [ 717.821183] [<ffffffff8023de88>] ? update_process_times+0x26/0x4b [ 717.821183] [<ffffffff80250535>] ? tick_sched_timer+0x76/0xa7 [ 717.821183] [<ffffffff802504bf>] ? tick_sched_timer+0x0/0xa7 [ 717.821183] [<ffffffff80249dba>] ? __run_hrtimer+0x55/0x92 [ 717.821183] [<ffffffff8024a7e7>] ? hrtimer_interrupt+0xe3/0x15d [ 717.821183] [<ffffffff8021c23c>] ? smp_apic_timer_interrupt+0x89/0xa9 [ 717.821183] [<ffffffff8020cbb6>] ? apic_timer_interrupt+0x66/0x70 [ 717.821183] <EOI> [<ffffffff805ca561>] ? __atomic_notifier_call_chain+0x0/0x83 [ 717.821183] [<ffffffff805ca561>] ? __atomic_notifier_call_chain+0x0/0x83 [ 717.821183] [<ffffffff80212280>] ? default_idle+0x27/0x3b [ 717.821183] [<ffffffff80212469>] ? c1e_idle+0xd4/0xd8 [ 717.821183] [<ffffffff8020abd1>] ? cpu_idle+0x7b/0xa1 [ 717.821183] [ 717.821183] [ 717.821183] Code: 48 8b 45 18 48 39 d0 79 09 48 89 d0 48 89 55 18 eb 04 48 89 43 18 fe 03 eb 1e b8 00 01 00 00 f0 66 0f c1 45 00 38 e0 74 07 f3 90 <8a> 45 00 eb f5 48 89 ef e8 5a fe ff ff fe 45 00 5b 5d 41 5c c3 [ 717.821183] ---[ end trace 32605f607ce8ad50 ]--- > Same issue or is this a separate problem with the watchdog?
Hmm, not sure. Does the system lockup when you disable nmi_watchdog ?
We want to be sure that this is not some watchdog weirdness.
Thanks,
tglx
oh...someone removed me from CC list, fixed. Joshua I don't see from this tread - did you tried to play with "idle=" boot param? Possible options are: poll,mwait,halt,nomwait. Wonder if it helps. Thomas how do you think? > ------- Comment #31 from gorcunov@gmail.com 2008-09-28 12:16 -------
> oh...someone removed me from CC list, fixed. Joshua I don't see from this
> tread
> - did you tried to play with "idle=" boot param? Possible options are:
> poll,mwait,halt,nomwait. Wonder if it helps. Thomas how do you think?
Nope. I've not fiddled with idle.
-J
--
Could you try please? Not sure if it help but I'm really messed now with how we get stuck in NMI watchdog in your case loosing any interrupt activity. It should not eat too much time of you to test it I hope. > ------- Comment #33 from gorcunov@gmail.com 2008-09-29 09:31 -------
> Could you try please? Not sure if it help but I'm really messed now with how
> we
> get stuck in NMI watchdog in your case loosing any interrupt activity. It
> should not eat too much time of you to test it I hope.
What is my best bet? idle=poll ?
Why not :) And give nomwait a chance too. > ------- Comment #35 from gorcunov@gmail.com 2008-09-29 13:24 -------
> Why not :) And give nomwait a chance too.
How do you find out what mode the kernel has decided to use on it's own?
Will nomwait screw with dynticks?
-J
--
Argh, got another oops. I'm going to try disabling the watchdog as Thomas suggested. <Sep/29 10:01 am>/dev/sda3: 39561[ 92.023748] ------------[ cut here ]------------ <Sep/29 10:01 am>7/4006240 files [ 92.034528] EXT3 FS on sda3, internal journal <Sep/29 10:01 am>(1.0% non-contig[ 92.023748] WARNING: at kernel/mutex.c:351 mutex_trylock+0x52/0x111() <Sep/29 10:01 am>uous), 1620981/8[ 92.023748] Modules linked in:000000 blocks <Sep/29 10:01 am> k8temp forcedeth i2c_nforce2 [ ok ] i2c_core <Sep/29 10:01 am> * Remounting tg3root filesystem libphyread/write ... e1000 xfs [ ok dm_snapshot ] <Sep/29 10:01 am> dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Sep/29 10:01 am>[ 92.023748] Pid: 0, comm: swapper Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2 <Sep/29 10:01 am>[ 92.023748] <Sep/29 10:01 am>[ 92.023748] Call Trace: <Sep/29 10:01 am>[ 92.023748] <IRQ> [<ffffffff80235361>] warn_on_slowpath+0x51/0x77 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff805c63de>] mutex_trylock+0x52/0x111 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235ef4>] printk+0x4e/0x56 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8025f3cf>] crash_kexec+0x17/0xef <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8040aa5a>] do_unblank_screen+0xd/0x10c <Sep/29 10:01 am>[ 92.023748] [<ffffffff803bbab9>] bust_spinlocks+0x15/0x30 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235260>] panic+0x8f/0x13f <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80272551>] softlockup_tick+0x19e/0x1ab <Sep/29 10:01 am>[ 92.023748] [<ffffffff8023de88>] update_process_times+0x26/0x4b <Sep/29 10:01 am>[ 92.023748] [<ffffffff80250535>] tick_sched_timer+0x76/0xa7 <Sep/29 10:01 am>[ 92.023748] [<ffffffff802504bf>] tick_sched_timer+0x0/0xa7 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80249dba>] __run_hrtimer+0x55/0x92 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8024a7e7>] hrtimer_interrupt+0xe3/0x15d <Sep/29 10:01 am>[ 92.023748] [<ffffffff8021c23c>] smp_apic_timer_interrupt+0x89/0xa9 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020cbb6>] apic_timer_interrupt+0x66/0x70 <Sep/29 10:01 am>[ 92.023748] <EOI> [<ffffffff80212395>] c1e_idle+0x0/0xd8 <Sep/29 10:01 am>[ 92.023748] [<ffffffff805ca561>] __atomic_notifier_call_chain+0x0/0x83 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80212280>] default_idle+0x27/0x3b <Sep/29 10:01 am>[ 92.023748] [<ffffffff80212469>] c1e_idle+0xd4/0xd8 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] <Sep/29 10:01 am>[ 92.023748] ---[ end trace 16fac595db54a2dd ]--- <Sep/29 10:01 am> * Using /etc/mo[ 92.023748] ------------[ cut here ]------------ <Sep/29 10:01 am>dules.autoload.d[ 92.023748] WARNING: at kernel/smp.c:332 smp_call_function_mask+0x37/0x1d6() <Sep/29 10:01 am>/kernel-2.6 as c[ 92.023748] Modules linked in:onfig: <Sep/29 10:01 am> * Loa k8tempding module smsc forcedeth37b787_wdt ... i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Sep/29 10:01 am>[ 92.023748] Pid: 0, comm: swapper Tainted: G W 2.6.27-rc5-22033-gd26acd9-dirty #2 <Sep/29 10:01 am>[ 92.023748] <Sep/29 10:01 am>[ 92.023748] Call Trace: <Sep/29 10:01 am>[ 92.023748] <IRQ> [<ffffffff80235361>] warn_on_slowpath+0x51/0x77 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80234f85>] print_oops_end_marker+0x9/0x20 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235366>] warn_on_slowpath+0x56/0x77 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80256c53>] smp_call_function_mask+0x37/0x1d6 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8025f3cf>] crash_kexec+0x17/0xef <Sep/29 10:01 am>[ 92.023748] [<ffffffff8025f49e>] crash_kexec+0xe6/0xef <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8021b596>] native_smp_send_stop+0x1a/0x26 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235266>] panic+0x95/0x13f <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80272551>] softlockup_tick+0x19e/0x1ab <Sep/29 10:01 am>[ 92.023748] [<ffffffff8023de88>] update_process_times+0x26/0x4b <Sep/29 10:01 am>[ 92.023748] [<ffffffff80250535>] tick_sched_timer+0x76/0xa7 <Sep/29 10:01 am>[ 92.023748] [<ffffffff802504bf>] tick_sched_timer+0x0/0xa7 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80249dba>] __run_hrtimer+0x55/0x92 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8024a7e7>] hrtimer_interrupt+0xe3/0x15d <Sep/29 10:01 am>[ 92.023748] [<ffffffff8021c23c>] smp_apic_timer_interrupt+0x89/0xa9 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020cbb6>] apic_timer_interrupt+0x66/0x70 <Sep/29 10:01 am>[ 92.023748] <EOI> [<ffffffff80212395>] c1e_idle+0x0/0xd8 <Sep/29 10:01 am>[ 92.023748] [<ffffffff805ca561>] __atomic_notifier_call_chain+0x0/0x83 <Sep/29 10:01 am>[ 92.023748] [<ffffffff80212280>] default_idle+0x27/0x3b <Sep/29 10:01 am>[ 92.023748] [<ffffffff80212469>] c1e_idle+0xd4/0xd8 <Sep/29 10:01 am>[ 92.023748] [<ffffffff8020abd1>] cpu_idle+0x7b/0xa1 <Sep/29 10:01 am>[ 92.023748] <Sep/29 10:01 am>[ 92.023748] ---[ end trace 16fac595db54a2dd ]--- It happened again without nmi_watchdog set. Booting 'linux 2.6.27-rc5 (netdev) genkerne[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #2 SMP Fri Sep 12 10:04:18 HST 2008dirty roo [ 0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8 [ 0.000000] KERNEL supported cpus:=0x2f6e10] [ 0.000000] Intel GenuineIntelrnel-x86_64-2.6.27-rc5-22033-gd26acd9-dirty [ 0.000000] AMD AuthenticAMD1e0850 bytes] [ 0.000000] Centaur CentaurHauls [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 00000000cff60000 (usable) [ 0.000000] BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data) [ 0.000000] BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS) [ 0.000000] BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved) [ 0.000000] BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) [ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) [ 0.000000] BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved) [ 0.000000] BIOS-e820: 0000000100000000 - 0000000230000000 (usable) [ 0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff [ 0.000000] init_memory_mapping [ 0.000000] last_map_addr: cff60000 end: cff60000 [ 0.000000] init_memory_mapping [ 0.000000] last_map_addr: 230000000 end: 230000000 [ 0.000000] RAMDISK: 37e0f000 - 37fef850 [ 0.000000] DMI present. [ 0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD ) [ 0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD XSDT 6040000 LTP 0) [ 0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S 6040000 PTL_ F4240) [ 0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA CK8 6040000 MSFT 3000000) [ 0.000000] ACPI: FACS CFF6AFC0, 0040 [ 0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD POWERNOW 6040000 AMD 1) [ 0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD HAMMER 6040000 AMD 1) [ 0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD $UCRTBL$ 6040000 PTL 1) [ 0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD MCFG 6040000 LTP 0) [ 0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD HPETTBL 6040000 LTP 1) [ 0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD APIC 6040000 LTP 0) [ 0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD $SBFTBL$ 6040000 LTP 1) [ 0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 0-a0000 [ 0.000000] SRAT: Node 0 PXM 0 100000-d0000000 [ 0.000000] SRAT: Node 0 PXM 0 100000000-130000000 [ 0.000000] SRAT: Node 1 PXM 1 130000000-230000000 [ 0.000000] Bootmem setup node 0 0000000000000000-0000000130000000 [ 0.000000] NODE_DATA [0000000000015680 - 000000000002067f] [ 0.000000] bootmap [0000000000021000 - 0000000000046fff] pages 26 [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] [ 0.000000] #2 [0000200000 - 0000f96910] TEXT DATA BSS ==> [0000200000 - 0000f96910] [ 0.000000] #3 [0037e0f000 - 0037fef850] RAMDISK ==> [0037e0f000 - 0037fef850] [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved ==> [000009dc00 - 0000100000] [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE ==> [0000008000 - 000000c000] [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE ==> [000000c000 - 0000011000] [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP ==> [0000011000 - 0000015680] [ 0.000000] Bootmem setup node 1 0000000130000000-0000000230000000 [ 0.000000] NODE_DATA [0000000130000000 - 000000013000afff] [ 0.000000] bootmap [000000013000b000 - 000000013002afff] pages 20 [ 0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE [ 0.000000] #2 [0000200000 - 0000f96910] TEXT DATA BSS [ 0.000000] #3 [0037e0f000 - 0037fef850] RAMDISK [ 0.000000] #4 [000009dc00 - 0000100000] BIOS reserved [ 0.000000] #5 [0000008000 - 000000c000] PGTABLE [ 0.000000] #6 [000000c000 - 0000011000] PGTABLE [ 0.000000] #7 [0000011000 - 0000015680] MEMNODEMAP [ 0.000000] Scan SMP from ffff880000000000 for 1024 bytes. [ 0.000000] Scan SMP from ffff88000009fc00 for 1024 bytes. [ 0.000000] Scan SMP from ffff8800000f0000 for 65536 bytes. [ 0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600 [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal 0x00100000 -> 0x00230000 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[4] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x000cff60 [ 0.000000] 0: 0x00100000 -> 0x00130000 [ 0.000000] 1: 0x00130000 -> 0x00230000 [ 0.000000] Detected use of extended apic ids on hypertransport bus [ 0.000000] Detected use of extended apic ids on hypertransport bus [ 0.000000] ACPI: PM-Timer IO Port: 0x1008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] Setting APIC routing to flat [ 0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000 [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] mapped APIC to ffffffffff5fc000 ( fee00000) [ 0.000000] mapped IOAPIC to ffffffffff5fb000 (00000000fec00000) [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000 [ 0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000 [ 0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000 [ 0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000 [ 0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000 [ 0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000 [ 0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000 [ 0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000 [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000 [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000 [ 0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000 [ 0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000 [ 0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000 [ 0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000) [ 0.000000] PERCPU: Allocating 51872 bytes of per cpu data [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 2039539 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8 [ 0.000000] Initializing CPU#0 [ 0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes) [ 0.000000] Extended CMOS year: 2000 [ 0.000000] TSC: PIT calibration confirmed by PMTIMER. [ 0.000000] TSC: using PIT calibration value [ 0.000000] Detected 2799.937 MHz processor. [ 0.003333] Console: colour VGA+ 80x25 [ 0.003333] console [tty0] enabled [ 0.003333] console [ttyS0] enabled [ 0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.003333] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.003333] ... MAX_LOCK_DEPTH: 48 [ 0.003333] ... MAX_LOCKDEP_KEYS: 8191 [ 0.003333] ... CLASSHASH_SIZE: 4096 [ 0.003333] ... MAX_LOCKDEP_ENTRIES: 8192 [ 0.003333] ... MAX_LOCKDEP_CHAINS: 16384 [ 0.003333] ... CHAINHASH_SIZE: 8192 [ 0.003333] memory used by lock dependency info: 3839 kB [ 0.003333] per task-struct memory footprint: 1920 bytes [ 0.003333] Checking aperture... [ 0.003333] No AGP bridge found [ 0.003333] Node 0: aperture @ 0 size 32 MB [ 0.003333] Your BIOS doesn't leave a aperture memory hole [ 0.003333] Please enable the IOMMU option in the BIOS setup [ 0.003333] This costs you 64 MB of RAM [ 0.003333] Mapping aperture over 65536 KB of RAM @ 20000000 [ 0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000 [ 0.003333] Memory: 8108304k/9175040k available (3897k kernel code, 279268k reserved, 2242k data, 676k init) [ 0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2 [ 0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.20 BogoMIPS (lpj=9333123) [ 0.006694] Security Framework initialized [ 0.010740] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.019174] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.025693] Mount-cache hash table entries: 256 [ 0.026995] Initializing cgroup subsys ns [ 0.030004] Initializing cgroup subsys cpuacct [ 0.033335] Initializing cgroup subsys memory [ 0.036675] Initializing cgroup subsys devices [ 0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) [ 0.043332] CPU: L2 Cache: 1024K (64 bytes/line) [ 0.046665] CPU 0/0 -> Node 0 [ 0.050000] CPU: Physical Processor ID: 0 [ 0.053331] CPU: Processor Core ID: 0 [ 0.056670] using C1E aware idle routine [ 0.060022] ACPI: Core revision 20080609 [ 0.070057] Getting VERSION: 80050010 [ 0.073331] Getting VERSION: 80050010 [ 0.076663] Getting ID: 0 [ 0.079463] Getting ID: ff000000 [ 0.079994] Getting LVT0: 10000 [ 0.079997] Getting LVT1: 10000 [ 0.083331] masked ExtINT on CPU#0 [ 0.086906] ENABLING IO-APIC IRQs [ 0.090269] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.128392] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 [ 0.134743] Using local APIC timer interrupts. [ 0.139993] Detected 12.499 MHz APIC timer. [ 0.143481] lockdep: fixing up alternatives. [ 0.146681] Booting processor 1/1 ip 6000 [ 0.003333] Initializing CPU#1 [ 0.003333] masked ExtINT on CPU#1 [ 0.003333] Calibrating delay using timer specific routine.. 5602.22 BogoMIPS (lpj=9333160) [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) [ 0.003333] CPU 1/1 -> Node 0 [ 0.003333] CPU: Physical Processor ID: 0 [ 0.003333] CPU: Processor Core ID: 1 [ 0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106 [ 0.243706] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 [ 0.253359] lockdep: fixing up alternatives. [ 0.256663] Booting processor 2/2 ip 6000 [ 0.003333] Initializing CPU#2 [ 0.003333] masked ExtINT on CPU#2 [ 0.003333] Calibrating delay using timer specific routine.. 5602.27 BogoMIPS (lpj=9333232) [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) [ 0.003333] CPU 2/2 -> Node 1 [ 0.003333] CPU: Physical Processor ID: 1 [ 0.003333] CPU: Processor Core ID: 0 [ 0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106 [ 0.353880] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 [ 0.363384] lockdep: fixing up alternatives. [ 0.366663] Booting processor 3/3 ip 6000 [ 0.003333] Initializing CPU#3 [ 0.003333] masked ExtINT on CPU#3 [ 0.003333] Calibrating delay using timer specific routine.. 5602.26 BogoMIPS (lpj=9333225) [ 0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) [ 0.003333] CPU: L2 Cache: 1024K (64 bytes/line) [ 0.003333] CPU 3/3 -> Node 1 [ 0.003333] CPU: Physical Processor ID: 1 [ 0.003333] CPU: Processor Core ID: 1 [ 0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106 [ 0.463866] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03 [ 0.473316] Brought up 4 CPUs [ 0.476464] Total of 4 processors activated (22408.96 BogoMIPS). [ 0.480095] net_namespace: 968 bytes [ 0.483490] xor: automatically using best checksumming function: generic_sse [ 0.503303] generic_sse: 8630.400 MB/sec [ 0.506635] xor: using function: generic_sse (8630.400 MB/sec) [ 0.510036] NET: Registered protocol family 16 [ 0.513587] No dock devices found. [ 0.516836] TOM: 00000000d0000000 aka 3328M [ 0.519976] TOM2: 0000000230000000 aka 8960M [ 0.523324] ACPI: bus type pci registered [ 0.526780] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4 [ 0.529969] PCI: MCFG area at e0000000 reserved in E820 [ 0.533407] PCI: Using MMCONFIG at e0000000 - e04fffff [ 0.536635] PCI: Using configuration type 1 for base access [ 0.551716] ACPI: Interpreter enabled [ 0.553302] ACPI: (supports S0 S1 S3 S4 S5) [ 0.559969] ACPI: Using IOAPIC for interrupt routing [ 0.573012] ACPI: PCI Root Bridge [PCI0] (0000:00) [ 0.573607] pci 0000:00:01.1: PME# supported from D3hot D3cold [ 0.576639] pci 0000:00:01.1: PME# disabled [ 0.580011] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.583303] pci 0000:00:02.0: PME# disabled [ 0.589979] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold [ 0.593303] pci 0000:00:02.1: PME# disabled [ 0.596867] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.599971] pci 0000:00:08.0: PME# disabled [ 0.603353] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.606637] pci 0000:00:09.0: PME# disabled [ 0.610001] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.613303] pci 0000:00:0a.0: PME# disabled [ 0.616665] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.619969] pci 0000:00:0d.0: PME# disabled [ 0.623331] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.626636] pci 0000:00:0f.0: PME# disabled [ 0.630232] pci 0000:00:06.0: transparent bridge [ 0.633303] PCI: bridge 0000:00:06.0 io port: [3000, 3fff] [ 0.636636] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff] [ 0.639969] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff] [ 0.643399] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff] [ 0.672620] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.680119] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23) [ 0.689632] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.697156] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.708161] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.719233] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23) [ 0.725213] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.732884] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.740492] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.751495] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23) [ 0.761253] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23) [ 0.770941] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23) [ 0.780630] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23) [ 0.790117] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.800492] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled. [ 0.811945] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23) [ 0.818230] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23) [ 0.827922] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23) [ 0.837749] Linux Plug and Play Support v0.97 (c) Adam Belay [ 0.840034] pnp: PnP ACPI init [ 0.843313] ACPI: bus type pnp registered [ 0.852025] pnp: PnP ACPI: found 14 devices [ 0.853303] ACPI: ACPI bus type pnp unregistered [ 0.856900] SCSI subsystem initialized [ 0.860410] usbcore: registered new interface driver usbfs [ 0.863368] usbcore: registered new interface driver hub [ 0.866740] usbcore: registered new device driver usb [ 0.870337] PCI: Using ACPI for IRQ routing [ 0.873307] testing the IO APIC....................... [ 0.876640] [ 0.880152] .................................... done. [ 0.896897] PCI-DMA: Disabling AGP. [ 0.900860] PCI-DMA: aperture base @ 20000000 size 65536 KB [ 0.903302] PCI-DMA: using GART IOMMU. [ 0.906637] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture [ 0.910474] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31 [ 0.917011] hpet0: 3 32-bit timers, 25000000 Hz [ 0.921064] ACPI: RTC can wake from S4 [ 0.933369] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved [ 0.941202] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved [ 0.949032] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved [ 0.956860] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved [ 0.964206] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved [ 0.971543] system 00:04: ioport range 0x1000-0x107f has been reserved [ 0.978269] system 00:04: ioport range 0x1080-0x10ff has been reserved [ 0.984993] system 00:04: ioport range 0x1400-0x147f has been reserved [ 0.991718] system 00:04: ioport range 0x1480-0x14ff has been reserved [ 0.998442] system 00:04: ioport range 0x1800-0x187f has been reserved [ 1.005175] system 00:04: ioport range 0x1880-0x18ff has been reserved [ 1.011904] system 00:04: ioport range 0x2440-0x247f has been reserved [ 1.018629] system 00:04: ioport range 0x2400-0x243f has been reserved [ 1.025360] system 00:07: ioport range 0x4d0-0x4d1 has been reserved [ 1.031914] system 00:07: ioport range 0xc05-0xc06 has been reserved [ 1.039645] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01 [ 1.045762] pci 0000:00:06.0: IO window: 0x3000-0x3fff [ 1.051273] pci 0000:00:06.0: MEM window: 0xd0100000-0xd01fffff [ 1.057572] pci 0000:00:06.0: PREFETCH window: 0x000000d8000000-0x000000dfffffff [ 1.065488] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02 [ 1.071606] pci 0000:00:0a.0: IO window: disabled [ 1.076683] pci 0000:00:0a.0: MEM window: 0xd0200000-0xd02fffff [ 1.082969] pci 0000:00:0a.0: PREFETCH window: 0x000000d1000000-0x000000d10fffff [ 1.090895] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03 [ 1.097008] pci 0000:00:0d.0: IO window: disabled [ 1.102085] pci 0000:00:0d.0: MEM window: disabled [ 1.107250] pci 0000:00:0d.0: PREFETCH window: disabled [ 1.112843] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04 [ 1.118959] pci 0000:00:0f.0: IO window: disabled [ 1.124028] pci 0000:00:0f.0: MEM window: disabled [ 1.129185] pci 0000:00:0f.0: PREFETCH window: disabled [ 1.134800] bus: 00 index 0 io port: [0, ffff] [ 1.139437] bus: 00 index 1 mmio: [0, ffffffffffffffff] [ 1.144853] bus: 01 index 0 io port: [3000, 3fff] [ 1.149749] bus: 01 index 1 mmio: [d0100000, d01fffff] [ 1.155080] bus: 01 index 2 mmio: [d8000000, dfffffff] [ 1.165290] bus: 01 index 3 io port: [0, ffff] [ 1.169925] bus: 01 index 4 mmio: [0, ffffffffffffffff] [ 1.175343] bus: 02 index 0 mmio: [0, 0] [ 1.179459] bus: 02 index 1 mmio: [d0200000, d02fffff] [ 1.184789] bus: 02 index 2 mmio: [d1000000, d10fffff] [ 1.190122] bus: 02 index 3 mmio: [0, 0] [ 1.194249] bus: 03 index 0 mmio: [0, 0] [ 1.198369] bus: 03 index 1 mmio: [0, 0] [ 1.202486] bus: 03 index 2 mmio: [0, 0] [ 1.206602] bus: 03 index 3 mmio: [0, 0] [ 1.210720] bus: 04 index 0 mmio: [0, 0] [ 1.214836] bus: 04 index 1 mmio: [0, 0] [ 1.218951] bus: 04 index 2 mmio: [0, 0] [ 1.223067] bus: 04 index 3 mmio: [0, 0] [ 1.227203] NET: Registered protocol family 2 [ 1.263618] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 1.273549] TCP established hash table entries: 524288 (order: 11, 8388608 bytes) [ 1.286620] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes) [ 1.296287] TCP: Hash tables configured (established 524288 bind 65536) [ 1.303127] TCP reno registered [ 1.313494] NET: Registered protocol family 1 [ 1.318167] checking if image is initramfs... it is [ 1.418688] Freeing initrd memory: 1922k freed [ 1.424357] Simple Boot Flag at 0x37 set to 0x80 [ 1.432006] audit: initializing netlink socket (disabled) [ 1.437664] type=2000 audit(1222726720.437:1): initialized [ 1.448905] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.459521] VFS: Disk quotas dquot_6.5.1 [ 1.463839] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.472140] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 1.479048] msgmni has been set to 15840 [ 1.483400] async_tx: api initialized (async) [ 1.487965] io scheduler noop registered [ 1.492088] io scheduler anticipatory registered [ 1.496908] io scheduler deadline registered [ 1.501569] io scheduler cfq registered (default) [ 1.506513] pci 0000:00:00.0: Enabling HT MSI Mapping [ 1.714230] pci 0000:00:05.0: Enabling HT MSI Mapping [ 1.719503] pci 0000:00:05.1: Enabling HT MSI Mapping [ 1.724776] pci 0000:00:05.2: Enabling HT MSI Mapping [ 1.730049] pci 0000:00:06.0: Enabling HT MSI Mapping [ 1.735322] pci 0000:00:08.0: Enabling HT MSI Mapping [ 1.740590] pci 0000:00:09.0: Enabling HT MSI Mapping [ 1.745858] pci 0000:00:0a.0: Enabling HT MSI Mapping [ 1.751129] pci 0000:00:0d.0: Enabling HT MSI Mapping [ 1.756398] pci 0000:00:0f.0: Enabling HT MSI Mapping [ 1.761885] pcieport-driver 0000:00:0a.0: found MSI capability [ 1.768257] pcieport-driver 0000:00:0d.0: found MSI capability [ 1.774568] pcieport-driver 0000:00:0f.0: found MSI capability [ 1.786055] input: Power Button (FF) as /class/input/input0 [ 1.791842] ACPI: Power Button (FF) [PWRF] [ 1.796247] input: Power Button (CM) as /class/input/input1 [ 1.802015] ACPI: Power Button (CM) [PWRB] [ 1.806763] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) [ 1.812828] processor ACPI0007:00: registered as cooling_device0 [ 1.819040] ACPI: Processor [C000] (supports 2 throttling states) [ 1.825531] processor ACPI0007:01: registered as cooling_device1 [ 1.831830] processor ACPI0007:02: registered as cooling_device2 [ 1.838137] processor ACPI0007:03: registered as cooling_device3 [ 1.911989] Non-volatile memory driver v1.2 [ 1.916534] Linux agpgart interface v0.103 [ 1.920923] Serial: 8250/16550 driver4 ports, IRQ sharing disabled [ 1.927505] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.934476] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.940585] Floppy drive(s): fd0 is 1.44M [ 4.956690] floppy0: no floppy controllers found [ 4.963824] brd: module loaded [ 4.968171] loop: module loaded [ 4.971518] tun: Universal TUN/TAP device driver, 1.6 [ 4.976759] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 4.983339] console [netcon0] enabled [ 4.987198] netconsole: network logging started [ 4.991927] Uniform Multi-Platform E-IDE driver [ 4.996788] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports [ 5.973537] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive [ 6.563432] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 [ 6.568257] ide1 at 0x170-0x177,0x376 on irq 15 [ 6.575067] hdb: ATAPI 24X CD-ROM drive, 256kB Cache [ 6.580450] Uniform CD-ROM driver Revision: 3.20 [ 6.595380] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006) [ 6.603180] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006) [ 6.610473] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008 [ 6.616664] Driver 'sd' needs updating - please use bus_type methods [ 6.623282] Driver 'sr' needs updating - please use bus_type methods [ 6.630795] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23 [ 6.636763] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23 [ 6.645374] sata_nv 0000:00:05.0: Using SWNCQ mode [ 6.650671] scsi0 : sata_nv [ 6.653847] scsi1 : sata_nv [ 6.657012] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23 [ 6.664351] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23 [ 7.000376] ata1: SATA link down (SStatus 0 SControl 300) [ 7.333704] ata2: SATA link down (SStatus 0 SControl 300) [ 7.339588] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22 [ 7.345557] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22 [ 7.354168] sata_nv 0000:00:05.1: Using SWNCQ mode [ 7.359414] scsi2 : sata_nv [ 7.362523] scsi3 : sata_nv [ 7.365685] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22 [ 7.373020] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22 [ 7.707000] ata3: SATA link down (SStatus 0 SControl 300) [ 8.040331] ata4: SATA link down (SStatus 0 SControl 300) [ 8.046191] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21 [ 8.052159] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21 [ 8.060776] sata_nv 0000:00:05.2: Using SWNCQ mode [ 8.066013] scsi4 : sata_nv [ 8.069132] scsi5 : sata_nv [ 8.072289] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21 [ 8.079622] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21 [ 8.413705] ata5: SATA link down (SStatus 0 SControl 300) [ 8.747037] ata6: SATA link down (SStatus 0 SControl 300) [ 8.752793] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7] [ 8.760551] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16) [ 8.768821] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177] [ 8.776567] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16) [ 8.784836] pata_amd 0000:00:04.0: no available native port [ 8.790918] Fusion MPT base driver 3.04.07 [ 8.795214] Copyright (c) 1999-2008 LSI Corporation [ 8.800298] Fusion MPT SPI Host driver 3.04.07 [ 8.805026] Fusion MPT SAS Host driver 3.04.07 [ 8.810358] ieee1394: raw1394: /dev/raw1394 device initialized [ 8.816877] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20 [ 8.822837] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20 [ 8.831550] ehci_hcd 0000:00:02.1: EHCI Host Controller [ 8.837215] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1 [ 8.844998] ehci_hcd 0000:00:02.1: debug port 1 [ 8.849747] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000 [ 8.863348] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 [ 8.871323] usb usb1: configuration #1 chosen from 1 choice [ 8.877185] hub 1-0:1.0: USB hub found [ 8.881145] hub 1-0:1.0: 10 ports detected [ 8.987275] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19 [ 8.993239] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19 [ 9.001945] ohci_hcd 0000:00:02.0: OHCI Host Controller [ 9.007531] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2 [ 9.015293] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000 [ 9.075474] usb usb2: configuration #1 chosen from 1 choice [ 9.081323] hub 2-0:1.0: USB hub found [ 9.085280] hub 2-0:1.0: 10 ports detected [ 9.190297] USB Universal Host Controller Interface driver v3.0 [ 9.196582] usbcore: registered new interface driver usblp [ 9.202265] Initializing USB Mass Storage driver... [ 9.207407] usbcore: registered new interface driver usb-storage [ 9.213610] USB Mass Storage support registered. [ 9.218573] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12 [ 9.473636] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.479008] mice: PS/2 mouse device common for all mice [ 9.603338] md: linear personality registered for level -1 [ 9.684166] md: raid0 personality registered for level 0 [ 9.689677] md: raid1 personality registered for level 1 [ 9.695181] md: raid10 personality registered for level 10 [ 9.754596] raid6: int64x1 2559 MB/s [ 9.814595] raid6: int64x2 3271 MB/s [ 9.874588] raid6: int64x4 3205 MB/s [ 9.934602] raid6: int64x8 2272 MB/s [ 9.994596] raid6: sse2x1 3909 MB/s [ 10.054585] raid6: sse2x2 5182 MB/s [ 10.114594] raid6: sse2x4 5221 MB/s [ 10.118545] raid6: using algorithm sse2x4 (5221 MB/s) [ 10.123791] md: raid6 personality registered for level 6 [ 10.129301] md: raid5 personality registered for level 5 [ 10.134807] md: raid4 personality registered for level 4 [ 10.140318] md: multipath personality registered for level -4 [ 10.151370] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com [ 10.160290] cpuidle: using governor ladder [ 10.164810] cpuidle: using governor menu [ 10.169007] usbcore: registered new interface driver usbhid [ 10.174781] usbhid: v2.6:USB HID core driver [ 10.179352] oprofile: using NMI interrupt. [ 10.184099] TCP cubic registered [ 10.187720] NET: Registered protocol family 10 [ 10.193358] IPv6 over IPv4 tunneling driver [ 10.198469] NET: Registered protocol family 17 [ 10.203498] RPC: Registered udp transport module. [ 10.208403] RPC: Registered tcp transport module. [ 10.213394] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00) [ 10.220006] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 [ 10.230248] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa [ 10.236103] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc [ 10.241962] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe [ 10.247733] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 [ 10.253583] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 [ 10.259441] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 [ 10.214172] powernow-k8: 0 : fid 0x14 (2800 MHz), vid 0x8 [ 10.271343] powernow-k8: 1 : fid 0x12 (2600 MHz), vid 0xa [ 10.277212] powernow-k8: 2 : fid 0x10 (2400 MHz), vid 0xc [ 10.283072] powernow-k8: 3 : fid 0xe (2200 MHz), vid 0xe [ 10.288844] powernow-k8: 4 : fid 0xc (2000 MHz), vid 0x10 [ 10.294703] powernow-k8: 5 : fid 0xa (1800 MHz), vid 0x10 [ 10.300567] powernow-k8: 6 : fid 0x2 (1000 MHz), vid 0x12 [ 10.307138] drivers/rtc/hctosys.c: unable to open rtc device (rtc0) [ 10.313682] Freeing unused kernel memory: 676k freed [ 10.319217] Write protecting the kernel read-only data: 5672k >> Loading modules :: Scanning for scsi_wait_scan...scsi_wait_scan loaded. :: Scanning for aic7xxx...aic7xxx loaded. :: Scanning for arcmsr...arcmsr loaded. :: Scanning for atp870u...atp870u loaded. :: Scanning for 3w-xxxx...3w-xxxx loaded. :: Scanning for 3w-9xxx...3w-9xxx loaded. :: Scanning for aacraid...aacraid loaded. :: Scanning for dm-mirror...dm-log, dm-mirror loaded. :: Scanning for dm-snapshot...dm-snapshot loaded. :: Scanning for xfs...xfs loaded. :: Scanning for e1000...e1000 loaded. :: Scanning for tg3...libphy, tg3 loaded. >> Activating mdev >> Determining root device... >> Mounting root... >> Booting (initramfs).. INIT: version 2.86 booting Gentoo Linux; http://www.gentoo.org/ Copyright 1999-2007 Gentoo Foundation; Distributed under the GPLv2 Press I to enter interactive boot mode * Mounting proc at /proc ... [ ok ] * Mounting sysfs at /sys ... [ ok ] * Mounting /dev for udev ... [ ok ] * Seeding /dev with needed nodes ... [ ok ] * Starting udevd ... [ ok ] * Populating /dev with existing devices through uevents ... [ ok ] * Letting udev process events ... * udev loading module forcedeth * udev loading module i2c_nforce2 * udev loading module k8temp [ ok ] * Finalizing udev configuration ... [ ok ] * Mounting devpts at /dev/pts ... [ ok ] * Checking root filesystem .../dev/sda3: clean, 395874/4006240 files, 1770311/8000000 blocks [ ok ] * Remounting root filesystem read/write ... [ ok ] * Using /etc/modules.autoload.d/kernel-2.6 as config: * Loading module smsc37b787_wdt ... [ ok ] * Autoloaded 1 module(s) * Setting up the Logical Volume Manager ... /dev/cdrom: open failed: Read-only file system Attempt to close device '/dev/cdrom' which is not open. /dev/cdrom: open failed: Read-only file system Attempt to close device '/dev/cdrom' which is not open. No volume groups found No volume groups found No volume groups found [ ok ] * Setting up dm-crypt mappings ... [ ok ] * Checking all filesystems .../dev/sda1: clean, 68/24480 files, 71784/97636 blocks ipp008.0 contains a file system with errors, check forced. ipp008.0: Inode 228356567 is in use, but has dtime set. FIXED. ipp008.0: Inode 228356567 has imagic flag set. ipp008.0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) * Fsck could not correct all errors, manual repair needed [ !! ] Give root password for maintenance (or type Control-D to continue): (none) ~ # fsck -ay -C -v /dev/sda4 fsck 1.40.4 (31-Dec-2007) e2fsck: Only one of the options -p/-a, -n or -y may be specified. (none) ~ # fsck -y -C -v /dev/sda4 fsck 1.40.4 (31-Dec-2007) e2fsck 1.40.4 (31-Dec-2007) ipp008.0 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes [12491.499178] Kernel panic - not syncing: softlockup: hung tasks - 6.3% [12491.499178] ------------[ cut here ]------------ [12491.499178] WARNING: at kernel/mutex.c:351 mutex_trylock+0x52/0x111() [12491.499178] Modules linked in: smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan [12491.499178] Pid: 303, comm: kswapd0 Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2 [12491.499178] [12491.499178] Call Trace: [12491.499178] <IRQ> [<ffffffff80235361>] warn_on_slowpath+0x51/0x77 [12491.499178] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 [12491.499178] [<ffffffff805c63de>] mutex_trylock+0x52/0x111 [12491.499178] [<ffffffff8025f3cf>] crash_kexec+0x17/0xef [12491.499178] [<ffffffff803bbab9>] bust_spinlocks+0x15/0x30 [12491.499178] [<ffffffff80235260>] panic+0x8f/0x13f [12491.499178] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 [12491.499178] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 [12491.499178] [<ffffffff80272551>] softlockup_tick+0x19e/0x1ab [12491.499178] [<ffffffff8023de88>] update_process_times+0x26/0x4b [12491.499178] [<ffffffff80250535>] tick_sched_timer+0x76/0xa7 [12491.499178] [<ffffffff802504bf>] tick_sched_timer+0x0/0xa7 [12491.499178] [<ffffffff80249dba>] __run_hrtimer+0x55/0x92 [12491.499178] [<ffffffff8024a7e7>] hrtimer_interrupt+0xe3/0x15d [12491.499178] [<ffffffff8021c23c>] smp_apic_timer_interrupt+0x89/0xa9 [12491.499178] [<ffffffff8020cbb6>] apic_timer_interrupt+0x66/0x70 [12491.499178] <EOI> [<ffffffff80266e5c>] res_counter_uncharge+0x1d/0x3f [12491.499178] [<ffffffff80252f1b>] lock_acquire+0x67/0x6d [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff805c7c79>] _spin_lock+0x29/0x34 [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff80281196>] shrink_page_list+0x440/0x57f [12491.499178] [<ffffffff80280ae2>] isolate_lru_pages+0x62/0x1f3 [12491.499178] [<ffffffff80281424>] shrink_inactive_list+0x133/0x3a0 [12491.499178] [<ffffffff80281798>] shrink_zone+0x107/0x12a [12491.499178] [<ffffffff802824e3>] kswapd+0x303/0x4ac [12491.499178] [<ffffffff80280c73>] isolate_pages_global+0x0/0x2f [12491.499178] [<ffffffff805c5f64>] thread_return+0x3e/0xa8 [12491.499178] [<ffffffff8024791a>] autoremove_wake_function+0x0/0x2e [12491.499178] [<ffffffff802821e0>] kswapd+0x0/0x4ac [12491.499178] [<ffffffff80247805>] kthread+0x47/0x76 [12491.499178] [<ffffffff80230223>] schedule_tail+0x27/0x5f [12491.499178] [<ffffffff8020ce09>] child_rip+0xa/0x11 [12491.499178] [<ffffffff8024767c>] kthreadd+0x167/0x18c [12491.499178] [<ffffffff802477be>] kthread+0x0/0x76 [12491.499178] [<ffffffff8020cdff>] child_rip+0x0/0x11 [12491.499178] [12491.499178] ---[ end trace 6a9aab8b65c1a995 ]--- [12491.499178] ------------[ cut here ]------------ [12491.499178] WARNING: at kernel/smp.c:332 smp_call_function_mask+0x37/0x1d6() [12491.499178] Modules linked in: smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan [12491.499178] Pid: 303, comm: kswapd0 Tainted: G W 2.6.27-rc5-22033-gd26acd9-dirty #2 [12491.499178] [12491.499178] Call Trace: [12491.499178] <IRQ> [<ffffffff80235361>] warn_on_slowpath+0x51/0x77 [12491.499178] [<ffffffff80234f85>] print_oops_end_marker+0x9/0x20 [12491.499178] [<ffffffff80235366>] warn_on_slowpath+0x56/0x77 [12491.499178] [<ffffffff80256c53>] smp_call_function_mask+0x37/0x1d6 [12491.499178] [<ffffffff8025f3cf>] crash_kexec+0x17/0xef [12491.499178] [<ffffffff8025f49e>] crash_kexec+0xe6/0xef [12491.499178] [<ffffffff8021b596>] native_smp_send_stop+0x1a/0x26 [12491.499178] [<ffffffff80235266>] panic+0x95/0x13f [12491.499178] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 [12491.499178] [<ffffffff80235816>] release_console_sem+0x3e/0x1a1 [12491.499178] [<ffffffff80272551>] softlockup_tick+0x19e/0x1ab [12491.499178] [<ffffffff8023de88>] update_process_times+0x26/0x4b [12491.499178] [<ffffffff80250535>] tick_sched_timer+0x76/0xa7 [12491.499178] [<ffffffff802504bf>] tick_sched_timer+0x0/0xa7 [12491.499178] [<ffffffff80249dba>] __run_hrtimer+0x55/0x92 [12491.499178] [<ffffffff8024a7e7>] hrtimer_interrupt+0xe3/0x15d [12491.499178] [<ffffffff8021c23c>] smp_apic_timer_interrupt+0x89/0xa9 [12491.499178] [<ffffffff8020cbb6>] apic_timer_interrupt+0x66/0x70 [12491.499178] <EOI> [<ffffffff80266e5c>] res_counter_uncharge+0x1d/0x3f [12491.499178] [<ffffffff80252f1b>] lock_acquire+0x67/0x6d [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff805c7c79>] _spin_lock+0x29/0x34 [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff802c4dc3>] try_to_free_buffers+0x51/0xa0 [12491.499178] [<ffffffff80281196>] shrink_page_list+0x440/0x57f [12491.499178] [<ffffffff80280ae2>] isolate_lru_pages+0x62/0x1f3 [12491.499178] [<ffffffff80281424>] shrink_inactive_list+0x133/0x3a0 [12491.499178] [<ffffffff80281798>] shrink_zone+0x107/0x12a [12491.499178] [<ffffffff802824e3>] kswapd+0x303/0x4ac [12491.499178] [<ffffffff80280c73>] isolate_pages_global+0x0/0x2f [12491.499178] [<ffffffff805c5f64>] thread_return+0x3e/0xa8 [12491.499178] [<ffffffff8024791a>] autoremove_wake_function+0x0/0x2e [12491.499178] [<ffffffff802821e0>] kswapd+0x0/0x4ac [12491.499178] [<ffffffff80247805>] kthread+0x47/0x76 [12491.499178] [<ffffffff80230223>] schedule_tail+0x27/0x5f [12491.499178] [<ffffffff8020ce09>] child_rip+0xa/0x11 [12491.499178] [<ffffffff8024767c>] kthreadd+0x167/0x18c [12491.499178] [<ffffffff802477be>] kthread+0x0/0x76 [12491.499178] [<ffffffff8020cdff>] child_rip+0x0/0x11 [12491.499178] [12491.499178] ---[ end trace 6a9aab8b65c1a995 ]--- > [ 0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty Can you please test 2.6.27-rc8 ? We need a common base to look at. Your network investigations are a separate issue. 2.6.27-rc5 is known to be problematic with C1E machines and we really need to know whether we catched all corner cases in mainline. > [12491.499178] Kernel panic - not syncing: softlockup: hung tasks - 6.3% Can you please disable the panic on softlockup ? CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 Crashing the machine does not help as we can not gather more information from the machine. Can you trigger it with fsck reproducible ? Thanks, tglx > How do you find out what mode the kernel has decided to use on it's own?
> Will nomwait screw with dynticks?
Boot log with "debug" option will print (or not in case of default idle) message. About dyntricks - not sure.
Reply-To: josh@hoblitt.com > ------- Comment #39 from tglx@linutronix.de 2008-09-30 02:03 ------- > > [ 0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty > > Can you please test 2.6.27-rc8 ? Yes. > We need a common base to look at. Your network investigations are a > separate issue. 2.6.27-rc5 is known to be problematic with C1E > machines and we really need to know whether we catched all corner > cases in mainline. I believe the fixes I need have been merged in from netdev-2.6. Without them the machine will experience other random failures that I believe had been obscuring this issue. > > [12491.499178] Kernel panic - not syncing: softlockup: hung tasks - 6.3% > > Can you please disable the panic on softlockup ? > > CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 Can do. > Crashing the machine does not help as we can not gather more > information from the machine. > > Can you trigger it with fsck reproducible ? Maybe? I got the panic on one of the machines again last night. It seems to come in waves on various different nodes and then go away for weeks/months. -J -- > > ------- Comment #39 from tglx@linutronix.de 2008-09-30 02:03 -------
> > > [ 0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty
> >
> > Can you please test 2.6.27-rc8 ?
>
> Yes.
I'm testing -rc8 w/o panic on softlockup on another machine right now but one
the machines running -rc5 from netdev-2.6 just had another oops that looks
different from the others.
[13444.209171] BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
[13444.212042] IP: [<ffffffff80218e16>] powernowk8_target+0x61d/0x857
[13444.212042] PGD 2179c3067 PUD 22b438067 PMD 0
[13444.212042] Oops: 0002 [1] SMP
[13444.212042] CPU 0
[13444.212042] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 forcedeth i2c_core k8temp tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
[13444.212042] Pid: 251, comm: kondemand/0 Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2
[13444.212042] RIP: 0010:[<ffffffff80218e16>] [<ffffffff80218e16>] powernowk8_target+0x61d/0x857
[13444.212042] RSP: 0018:ffff88012fbebdc0 EFLAGS: 00010246
[13444.212042] RAX: 0000000000000000 RBX: ffff88012e502600 RCX: 0000000000000002
[13444.212042] RDX: 0000000000000008 RSI: 0000000000000002 RDI: ffff88012e502600
[13444.212042] RBP: 000000000000000c R08: 0000000000000008 R09: 0000000000000190
[13444.212042] R10: ffff88012fbebd50 R11: ffffffff803bac5e R12: 0000000000000002
[13444.212042] R13: 0000000000000012 R14: 0000000000000008 R15: 0000000000000000
[13444.212042] FS: 0000000042427950(0000) GS:ffffffff80858e00(0000) knlGS:0000000000000000
[13444.212042] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[13444.212042] CR2: 0000000000000002 CR3: 00000002179c2000 CR4: 00000000000006e0
[13444.212042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[13444.212042] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[13444.212042] Process kondemand/0 (pid: 251, threadinfo ffff88012fbea000, task ffff88012fbe3c40)
[13444.212042] Stack: 0000000000000002 ffff880000000000 ffff88012e459600 0000001400000282
[13444.212042] 002ab98000000001 ffff8800000f4240 0000000000000001 000000062e459600
[13444.212042] 0000000000000000 0000000000000046 ffff88012e459600 000000000000007e
[13444.212042] Call Trace:
[13444.212042] [<ffffffff8050d244>] ? do_dbs_timer+0x1e0/0x24f
[13444.212042] [<ffffffff8050d064>] ? do_dbs_timer+0x0/0x24f
[13444.212042] [<ffffffff8050d064>] ? do_dbs_timer+0x0/0x24f
[13444.212042] [<ffffffff80244394>] ? run_workqueue+0xed/0x1ed
[13444.212042] [<ffffffff8024433e>] ? run_workqueue+0x97/0x1ed
[13444.212042] [<ffffffff80244ef7>] ? worker_thread+0xd8/0xe3
[13444.212042] [<ffffffff8024791a>] ? autoremove_wake_function+0x0/0x2e
[13444.212042] [<ffffffff80244e1f>] ? worker_thread+0x0/0xe3
[13444.212042] [<ffffffff80247805>] ? kthread+0x47/0x76
[13444.212042] [<ffffffff80230223>] ? schedule_tail+0x27/0x5f
[13444.212042] [<ffffffff8020ce09>] ? child_rip+0xa/0x11
[13444.212042] [<ffffffff8024767c>] ? kthreadd+0x167/0x18c
[13444.212042] [<ffffffff802477be>] ? kthread+0x0/0x76
[13444.212042] [<ffffffff8020cdff>] ? child_rip+0x0/0x11
[13444.212042]
[13444.212042]
[13444.212042] Code: c7 f7 31 6f 80 e8 b1 d0 01 00 e9 4c 01 00 00 8b 53 28 41 39 d6 74 0e 44 89 f6 48 c7 c7 42 32 6f 80 31 c0 eb e0 45 89 f0 44 89 e1 <48> c7 c2 7a 32 6f 80 48 c7 c6 f7 2b 6f 80 bf 02 00 00 00 31 c0
[13444.212042] RIP [<ffffffff80218e16>] powernowk8_target+0x61d/0x857
[13444.212042] RSP <ffff88012fbebdc0>
[13444.212042] CR2: 0000000000000002
[13444.530077] ---[ end trace 7cca0ea284e8104a ]---
Lets wait for your -rc8 experiments results. Hi Joshua, any news from -rc8? -rc8 proved itself stable for a few days on two test nodes and it's being rolled out onto 32 nodes production nodes as I type this. The bug strikes somewhat randomly so it'll take time for it to pop up in -rc8 if it's still there. Heavy jobs will be running over the weekend and should trip this issues if it's still in there... Thanks a lot for testing, Joshua! Report us the results in any case please. Hi Joshua, did -rc8 prove to be more/less stable for you? > ------- Comment #47 from gorcunov@gmail.com 2008-10-08 09:56 -------
> Hi Joshua, did -rc8 prove to be more/less stable for you?
It's not clear yet. We've had two random deadlocks on different
machines running -rc8. Unforunately, neither machine had a kernel trace
on the console.
-J
--
We just had the 3 or 4th -rc8 crash but at least this one had something on the console. <Oct/03 08:45 am>This is ipp016.unknown_domain (Linux x86_64 2.6.27-rc8-00016-gd3a47e8) 10:04:01 <Oct/08 02:25 pm>ipp016 login: [452552.081281] do_IRQ: 2.179 No irq handler for vector <Oct/08 02:29 pm>[452761.138609] BUG: spinlock lockup on CPU#2, perl/5830, ffff88022e412e58 <Oct/08 02:29 pm>[452761.646588] BUG: spinlock lockup on CPU#1, kswapd0/303, ffff88022e412e58 <Oct/08 02:29 pm>[452761.930596] BUG: spinlock lockup on CPU#0, sshd/1833, ffff88022e412e58 Thanks Joshua for report -- will check today evening. > <Oct/08 02:25 pm>ipp016 login: [452552.081281] do_IRQ: 2.179 No irq handler
> for
> vector
Hmm, this one is odd and might be the root of all evil. Have you seen that before ?
Thanks,
tglx
It seems some PCI-MSI doesn't have a vector assigned. Hmm.. What /proc/interrupt says on this machine? On Thu, Oct 09, 2008 at 08:06:34AM -0700, bugme-daemon@bugzilla.kernel.org wrote: > ------- Comment #52 from gorcunov@gmail.com 2008-10-09 08:06 ------- > It seems some PCI-MSI doesn't have a vector assigned. Hmm.. What > /proc/interrupt says on this machine? The sata_nv driver isn't it use. The only PCI[e] devices that see real use are forcedeth.c/eth0 and arcmsr. forcedeth.c has MSI turned off as there has been suspecions about MSI and this driver in the past. The exact options are "options forcedeth msi=0 msix=0 max_interrupt_work=30". PU0 CPU1 CPU2 CPU3 0: 43 0 2 373 IO-APIC-edge timer 1: 0 0 0 2 IO-APIC-edge i8042 4: 0 0 2 722 IO-APIC-edge serial 7: 1 0 0 0 IO-APIC-edge 9: 0 0 0 0 IO-APIC-fasteoi acpi 14: 0 1 0 198 IO-APIC-edge ide0 15: 0 0 0 0 IO-APIC-edge ide1 17: 347 852881 114490 163953647 IO-APIC-fasteoi eth0 18: 76 35497 59522 1300569 IO-APIC-fasteoi arcmsr 19: 0 0 0 0 IO-APIC-fasteoi ohci_hcd:usb2 20: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1 21: 0 0 0 0 IO-APIC-fasteoi sata_nv 22: 0 0 0 0 IO-APIC-fasteoi sata_nv 23: 0 0 0 0 IO-APIC-fasteoi sata_nv NMI: 0 0 0 0 Non-maskable interrupts LOC: 6977995 3235603 15722263 4343273 Local timer interrupts RES: 201328 208390 181427 128665 Rescheduling interrupts CAL: 717 945 2009 11092 function call interrupts TLB: 9367 14367 2038 1608 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 1 Reply-To: josh@hoblitt.com On Thu, Oct 09, 2008 at 02:21:39AM -0700, bugme-daemon@bugzilla.kernel.org wrote: > ------- Comment #51 from tglx@linutronix.de 2008-10-09 02:21 ------- > > <Oct/08 02:25 pm>ipp016 login: [452552.081281] do_IRQ: 2.179 No irq handler > for > > vector > > Hmm, this one is odd and might be the root of all evil. Have you seen that > before ? Yes, we've seen it before. I can find several examples of it being printed onto the console before a softlockup backtrace. Eg. -- [11532.103605] do_IRQ: 0.175 No irq handler for vector <Sep/11 12:13 pm>[11532.103613] do_IRQ: 2.175 No irq handler for vector <Sep/11 12:13 pm>[11532.103617] do_IRQ: 1.175 No irq handler for vector <Sep/11 12:14 pm>[11560.779989] do_IRQ: 0.179 No irq handler for vector -- -J -- I think it's time to ask forcedeth people about it. No sure who exactly to be asked. I've requested that one of the forcedeth devs consider that possibility in bug #9047. We hit the do_IRQ bug again. Any idea on how to debug this? <Oct/14 07:14 pm>ipp018 login: [969147.235296] do_IRQ: 1.179 No irq handler for vector <Oct/14 07:14 pm>[969148.722531] do_IRQ: 1.187 No irq handler for vector <Oct/14 07:14 pm>[969158.179808] do_IRQ: 0.187 No irq handler for vector Joshua I think I could take a look on holidays only (too busy now). Anyway -- it would be great if you could turn the modules you mentoined (especially forcedeth) off and see if do_IRQ fails since I have a suspicious it's NIC problem. We had another 4 crashes over night. 2 with nothing on the console, 1 just saying that the kernel was panicing, and this:
<Oct/15 05:38 pm>ipp014 login: [823902.827952] BUG: spinlock wrong owner on CPU#2, ppImage/27000
<Oct/15 05:38 pm>[823902.828447] lock: ffffffff80f425b8, .magic: dead4ead, .owner: ppImage/27000, .owner_cpu: 2
<Oct/15 05:38 pm>[823902.828447] Pid: 27000, comm: ppImage Not tainted 2.6.27-rc8-00016-gd3a47e8 #1
<Oct/15 05:38 pm>[823902.828447]
<Oct/15 05:38 pm>[823902.828447] Call Trace:
<Oct/15 05:38 pm>[823902.828447] [<ffffffff803bdf0e>] _raw_spin_unlock+0x51/0x80
<Oct/15 05:38 pm>[823902.828447] [<ffffffff805c82f8>] _spin_unlock_irqrestore+0x27/0x2e
<Oct/15 05:38 pm>[823902.828447] [<ffffffff802a27af>] __mem_cgroup_uncharge_common+0xcb/0x107
<Oct/15 05:38 pm>[823902.828447] [<ffffffff8028d642>] page_remove_rmap+0x108/0x122
<Oct/15 05:38 pm>[823902.828447] [<ffffffff80285e23>] unmap_vmas+0x47f/0x7ec
<Oct/15 05:38 pm>[823902.828447] [<ffffffff80289afb>] unmap_region+0xb3/0x126
<Oct/15 05:38 pm>[823902.828447] [<ffffffff8028a81b>] do_munmap+0x1f3/0x270
<Oct/15 05:38 pm>[823902.828447] [<ffffffff805c7ae6>] __down_write_nested+0x34/0x9e
<Oct/15 05:38 pm>[823902.828447] [<ffffffff8028b715>] sys_munmap+0x40/0x5a
<Oct/15 05:38 pm>[823902.828447] [<ffffffff8020be2b>] system_call_fastpath+0x16/0x1b
<Oct/15 05:38 pm>[823902.828447]
<Oct/15 05:47 pm>[824473.019627] BUG: unable to handle kernel NULL pointer dereference at 0000000000000158
<Oct/15 05:47 pm>[824473.021724] IP: [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
<Oct/15 05:47 pm>[824473.021724] PGD 21fc8b067 PUD 22e41c067 PMD 0
<Oct/15 05:47 pm>[824473.021724] Oops: 0000 [1] SMP
<Oct/15 05:47 pm>[824473.021724] CPU 2
<Oct/15 05:47 pm>[824473.021724] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Oct/15 05:47 pm>[824473.021724] Pid: 27703, comm: ppImage Not tainted 2.6.27-rc8-00016-gd3a47e8 #1
<Oct/15 05:47 pm>[824473.021724] RIP: 0010:[<ffffffff8022b04d>] [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
<Oct/15 05:47 pm>[824473.021724] RSP: 0018:ffff880223d69d18 EFLAGS: 00010046
<Oct/15 05:47 pm>[824473.021724] RAX: 000001e2b8508e7b RBX: ffff88021fc79e40 RCX: 0000000000000000
<Oct/15 05:47 pm>[824473.021724] RDX: ffff88013624a100 RSI: ffff88013624a1c8 RDI: ffff880223c09010
<Oct/15 05:47 pm>[824473.021724] RBP: ffff880223d69d48 R08: 0000000000000000 R09: ffff88013624a1c8
<Oct/15 05:47 pm>[824473.021724] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
<Oct/15 05:47 pm>[824473.021724] R13: 0002edda956529cd R14: ffff88013624a100 R15: 0000000000000000
<Oct/15 05:47 pm>[824473.021724] FS: 00007f83fbab56f0(0000) GS:ffff88022fa0d300(0000) knlGS:0000000000000000
<Oct/15 05:47 pm>[824473.021724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<Oct/15 05:47 pm>[824473.021724] CR2: 0000000000000158 CR3: 000000021e4ff000 CR4: 00000000000006e0
<Oct/15 05:47 pm>[824473.021724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Oct/15 05:47 pm>[824473.021724] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7<Oct/15 05:47 pm>0
<Oct/15 05:47 pm>[824473.021724] Process ppImage (pid: 27703, threadinfo ffff880223d68000, task ffff8800a6bb8f10)
<Oct/15 05:47 pm>[824473.021724] Stack: ffff88013624a100 ffff88013624a100 ffffffff805dc840 ffff880223d69d88
<Oct/15 05:47 pm>[824473.021724] 0000000000000000 000000010ebcc2ea ffff880223d69dd8 ffffffff805c5f2a
<Oct/15 05:47 pm>[824473.021724] 0000000000000000 ffffffff805567ed ffff8800a6bb8f10 0000000000000000
<Oct/15 05:47 pm>[824473.021724] Call Trace:
<Oct/15 05:47 pm>[824473.021724] [<ffffffff805c5f2a>] schedule+0x434/0x7e1
<Oct/15 05:47 pm>[824473.021724] [<ffffffff805567ed>] ? tcp_connect+0x319/0x322
<Oct/15 05:47 pm>[824473.021724] [<ffffffff8055974b>] ? tcp_v4_connect+0x3c1/0x422
<Oct/15 05:47 pm>[824473.021724] [<ffffffff805c64ab>] schedule_timeout+0x1e/0xad
<Oct/15 05:47 pm>[824473.021724] [<ffffffff805201cd>] ? release_sock+0x2f/0xd5
<Oct/15 05:47 pm>[824473.021724] [<ffffffff80564a0f>] ? inet_stream_connect+0x137/0x241
<Oct/15 05:47 pm>[824473.021724] [<ffffffff80247b2a>] ? autoremove_wake_function+0x0/0x2e
<Oct/15 05:47 pm>[824473.021724] [<ffffffff8051eb3b>] ? sys_connect+0x6c/0x9c
<Oct/15 05:47 pm>[824473.021724] [<ffffffff802a6290>] ? fget+0xb6/0xbe
<Oct/15 05:47 pm>[824473.021724] [<ffffffff805c7c2a>] ? lockdep_sys_exit_thunk+0x35/0x67
<Oct/15 05:47 pm>[824473.021724] [<ffffffff8039c9ac>] ? cap_file_fcntl+0x0/0x3
<Oct/15 05:47 pm>[824473.021724] [<ffffffff802a61da>] ? fget+0x0/0xbe
<Oct/15 05:47 pm>[824473.021724] [<ffffffff8020be2b>] ? system_call_fastpath+0x16/0x1b
<Oct/15 05:47 pm>[824473.021724]
<Oct/15 05:47 pm>[824473.021724]
<Oct/15 05:47 pm>[824473.021724] Code: ec e8 52 f5 ff ff 49 39 c4 76 11 49 8b 86 30 08 00 00 4d 8d 67 f0 48 89 43 28 eb 04 4c 8b 63 60 4c 89 e6 48 89 df e8 6a fa ff ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 9f 48 8b 7d d0 49 83 ec 38
<Oct/15 05:47 pm>[824473.021724] RIP [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
<Oct/15 05:47 pm>[824473.021724] RSP <ffff880223d69d18>
<Oct/15 05:47 pm>[824473.021724] CR2: 0000000000000158
<Oct/15 05:47 pm>[824473.021724] ---[ end trace 0dbe4de46ac70cd0 ]---
<Oct/15 05:50 pm>[824473.077977] BUG: spinlock lockup on CPU#1, swapper/0, ffff88013624a100
<Oct/15 05:50 pm>[824473.077977] Pid: 0, comm: swapper Tainted: G D 2.6.27-rc8-00016-gd3a47e8 #1
<Oct/15 05:50 pm>[824473.077977]
<Oct/15 05:50 pm>[824473.077977] Call Trace:
<Oct/15 05:50 pm>[824473.077977] <IRQ> [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
<Oct/15 05:50 pm>[824473.077977] [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80231c78>] tg_shares_up+0x0/0x195
<Oct/15 05:50 pm>[824473.124229] BUG: spinlock lockup on CPU#0, ppImage/27409, ffff88013624a100
<Oct/15 05:50 pm>[824473.124229] Pid: 27409, comm: ppImage Tainted: G D 2.6.27-rc8-00016-gd3a47e8 #1
<Oct/15 05:50 pm>[824473.124229]
<Oct/15 05:50 pm>[824473.124229] Call Trace:
<Oct/15 05:50 pm>[824473.124229] <IRQ> [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
<Oct/15 05:50 pm>[824473.124229] [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80231c78>] tg_shares_up+0x0/0x195
<Oct/15 05:50 pm>[824473.124229] [<ffffffff802296d8>] tg_nop+0x0/0x6
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8022e3c2>] try_to_wake_up+0xb0/0x238
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80229d46>] __wake_up_common+0x41/0x74
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8022a8d2>] __wake_up_sync+0x3a/0x56
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80521e4a>] sock_def_readable+0x3d/0x6a
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80553256>] tcp_rcv_established+0x60f/0x8e5
<Oct/15 05:50 pm>[824473.124229] [<ffffffff805586a3>] tcp_v4_do_rcv+0x2c/0x1b7
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80559e93>] tcp_v4_rcv+0x263/0x654
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8055a231>] tcp_v4_rcv+0x601/0x654
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80540d01>] ip_local_deliver+0xcb/0x164
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80540c91>] ip_local_deliver+0x5b/0x164
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80541294>] ip_rcv+0x4fa/0x552
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8052904b>] netif_receive_skb+0x24f/0x298
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80528f03>] netif_receive_skb+0x107/0x298
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8052b795>] process_backlog+0x71/0xbe
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8052b07e>] net_rx_action+0xc5/0x1fb
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8052b010>] net_rx_action+0x57/0x1fb
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8021d70b>] ack_apic_level+0x12/0xd3
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8020d17c>] call_softirq+0x1c/0x28
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8020e82a>] do_softirq+0x2c/0x68
<Oct/15 05:50 pm>[824473.124229] [<ffffffff80239cd6>] irq_exit+0x3f/0x85
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8020e9ca>] do_IRQ+0x13c/0x15e
<Oct/15 05:50 pm>[824473.124229] [<ffffffff8020c3f1>] ret_from_intr+0x0/0xa
<Oct/15 05:50 pm>[824473.124229] <EOI>
<Oct/15 05:50 pm>[824473.329415] BUG: spinlock lockup on CPU#2, ppImage/27703, ffff88013624a100
<Oct/15 05:50 pm>[824473.329415] Pid: 27703, comm: ppImage Tainted: G D 2<Oct/15 05:50 pm>d3a47e8 #1
<Oct/15 05:50 pm>[824473.329415]
<Oct/15 05:50 pm>[824473.329415] Call Trace:
<Oct/15 05:50 pm>[824473.329415] <IRQ> [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80232437>] scheduler_tick+0x47/0x1b8
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8023e0b1>] update_process_times+0x3f/0x4b
<Oct/15 05:50 pm>[824473.329415] [<ffffffff802506fa>] tick_sched_timer+0x76/0xa7
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80250684>] tick_sched_timer+0x0/0xa7
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80249fce>] __run_hrtimer+0x5a/0x97
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8024a9fb>] hrtimer_interrupt+0xe3/0x15d
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8021c29c>] smp_apic_timer_interrupt+0x89/0xa9
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8020cbc6>] apic_timer_interrupt+0x66/0x70
<Oct/15 05:50 pm>[824473.329415] <EOI> [<ffffffff80241808>] exit_signals+0xf4/0x10c
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c81ee>] _spin_unlock_irq+0x21/0x22
<Oct/15 05:50 pm>[824473.329415] [<ffffffff802417c6>] exit_signals+0xb2/0x10c
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80238123>] do_exit+0xd6/0x778
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c8851>] oops_begin+0x0/0x8c
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805ca732>] do_page_fault+0x73c/0x7f5
<Oct/15 05:50 pm>[824473.329415] [<ffffffff803b457d>] __next_cpu+0x19/0x26
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c844d>] error_exit+0x0/0x9a
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c5f2a>] schedule+0x434/0x7e1
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805567ed>] tcp_connect+0x319/0x322
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8055974b>] tcp_v4_connect+0x3c1/0x422
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c64ab>] schedule_timeout+0x1e/0xad
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805201cd>] release_sock+0x2f/0xd5
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80564a0f>] inet_stream_connect+0x137/0x241
<Oct/15 05:50 pm>[824473.329415] [<ffffffff80247b2a>] autoremove_wake_function+0x0/0x2e
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8051eb3b>] sys_connect+0x6c/0x9c
<Oct/15 05:50 pm>[824473.329415] [<ffffffff802a6290>] fget+0xb6/0xbe
<Oct/15 05:50 pm>[824473.329415] [<ffffffff805c7c2a>] lockdep_sys_exit_thunk+0x35/0x67
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8039c9ac>] cap_file_fcntl+0x0/0x3
<Oct/15 05:50 pm>[824473.329415] [<ffffffff802a61da>] fget+0x0/0xbe
<Oct/15 05:50 pm>[824473.329415] [<ffffffff8020be2b>] system_call_fastpath+0x16/0x1b
<Oct/15 05:50 pm>[824473.329415]
<Oct/15 05:50 pm>[824473.077977] [<ffffffff802296d8>] tg_nop+0x0/0x6
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8022e6b0>] rebalance_domains+0x13b/0x3f7
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80231990>] run_rebalance_domains+0x42/0xd8
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8020d17c>] call_softirq+0x1c/0x28
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8020e82a>] do_softirq+0x2c/0x68
<Oct/15 05:50 pm>[824473.077977] [<ffffffff80239cd6>] irq_exit+0x3f/0x85
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8021c2a1>] smp_apic_timer_interrupt+0x8e/0xa9
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8020cbc6>] apic_timer_interrupt+0x66/0x70
<Oct/15 05:50 pm>[824473.077977] <EOI> [<ffffffff802123da>] c1e_idle+0x0/0xe9
<Oct/15 05:50 pm>[824473.077977] [<ffffffff805ca841>] __atomic_notifier_call_chain+0x0/0x83
<Oct/15 05:50 pm>[824473.077977] [<ffffffff802122c5>] default_idle+0x27/0x3b
<Oct/15 05:50 pm>[824473.077977] [<ffffffff802124bf>] c1e_idle+0xe5/0xe9
<Oct/15 05:50 pm>[824473.077977] [<ffffffff8020abde>] cpu_idle+0x88/0xae
<Oct/15 05:50 pm>[824473.077977]
<Oct/15 05:51 pm>[824473.022141] BUG: spinlock lockup on CPU#3, ppImage/27440, ffff88013624a100
<Oct/15 05:51 pm>[824473.022141] Pid: 27440, comm: ppImage Tainted: G D 2.6.27-rc8-00016-gd3a47e8 #1
<Oct/15 05:51 pm>[824473.022141]
<Oct/15 05:51 pm>[824473.022141] Call Trace:
<Oct/15 05:51 pm>[824473.022141] <IRQ> [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
<Oct/15 05:51 pm>[824473.022141] [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80231c78>] tg_shares_up+0x0/0x195
<Oct/15 05:51 pm>[824473.022141] [<ffffffff802296d8>] tg_nop+0x0/0x6
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8022e3c2>] try_to_wake_up+0xb0/0x238
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80563ff6>] inet_sk_rebuild_header+0x19/0x31a
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80247b33>] autoremove_wake_function+0x9/0x2e
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80229d46>] __wake_up_common+0x41/0x74
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8022a926>] __wake_up+0x38/0x4f
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80520bb0>] sock_def_wakeup+0x3c/0x48
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8055b670>] tcp_init_congestion_control+0x19/0xc5
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8055250a>] tcp_rcv_state_process+0x2eb/0xa28
<Oct/15 05:51 pm>[824473.022141] [<ffffffff805587d8>] tcp_v4_do_rcv+0x161/0x1b7
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80559e93>] tcp_v4_rcv+0x263/0x654
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8055a231>] tcp_v4_rcv+0x601/0x654
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80540d01>] ip_local_deliver+0xcb/0x164
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80540c91>] ip_local_deliver+0x5b/0x164
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80541294>] ip_rcv+0x4fa/0x552
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8052904b>] netif_receive_skb+0x24f/0x298
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80528f03>] netif_receive_skb+0x107/0x298
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8052b795>] process_backlog+0x71/0xbe
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8052b07e>] net_rx_action+0xc5/0x1fb
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8052b010>] net_rx_action+0x57/0x1fb
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8021d70b>] ack_apic_level+0x12/0xd3
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8020d17c>] call_softirq+0x1c/0x28
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8020e82a>] do_softirq+0x2c/0x68
<Oct/15 05:51 pm>[824473.022141] [<ffffffff80239cd6>] irq_exit+0x3f/0x85
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8020e9ca>] do_IRQ+0x13c/0x15e
<Oct/15 05:51 pm>[824473.022141] [<ffffffff8020c3f1>] ret_from_intr+0x0/0xa
<Oct/15 05:51 pm>[824473.022141] <EOI>
Is it worth while to try disabling CONFIG_PCI_MSI ? iirc you were turning msi off with boot option so I'm not sure but you could try (to be sure). I think analyzing the dump above could help -- will try on holidays. Thanks! heh -- your OOPs looks similar to http://lkml.org/lkml/2008/5/13/212 checking... I mean having CR2: 0000000000000158 is quite strange. Do you have any KVM related stuff compiled in? It's compiled in but not in use. I suppose it will be eliminated if CONFIG_SCHED_GROUP disabled. It seems to be CFS related in which area I have no experience. Actually Dmitry suggested a debugging patch for this case http://lkml.org/lkml/2008/5/14/91 so you may try it (from what I've read this bug is still actual for mainline) Removing KVM isn't a big deal as we're not currently using it but we would feel losing CONFIG_SCHED_GROUP. We are running .27-rc8 from mainland now since what we needed from netdev-2.6 has been merged. If this is indeed a CFS bug, I'd like to hear Ingo's or Thomoas' thoughts on it before trying that patch. It could be not just CFS related but complex of CFS and hrtimers and hotplug. The Dmitry's patch is just to show CFS status if pick_next_task_fair fails. Ingo, Thomas? I've asked Peter Zijlkstra about probability of NULL deref in pick_next_task_fair which is not supposed to happen. So I don't know how it happens. My only suspicious is that happens like a side effect of some hotplug problem. Maybe a cpu core goes to sleep but don't update working structures and that leads to this NULL deref. Not sure Joshua. I wish I could help but it seems my knowledge is obviously not enough :( Hi Joshua, any news from practice? Same bugs? Did you try new shining kernel maybe? Joshua could you confirm if the problem is still here? I know it's boring -- but maybe you could try latest kernel (2.6.28-rc2)? The problem is still there in .27. I'm ok with trying to new kernels but it's takes several days to roll one out to the production cluster so I'd rather be testing some specific changeset that might address the issue. Joshua does it mean that problem is in picking new task? I mean -- the OOPs message is the same as last you posted (comment #59)? Joshua are you here? :) What is the most reproductible problem? The same as #59? Handled-By : Cyrill Gorcunov <gorcunov@gmail.com> Created attachment 18855 [details]
kernel panics from 2008-10-17 -> 2008-11-13
I'm sorry for the long lag in any response, I was out of the office for ~3 weeks. Summary of the last several weeks: We are still running 2.6.27-rc8 on 32 nodes in production. In the time period 2008-10-17 -> 2008-11-13 there were ~12 kernel panics with a backtrace (just added to the bug as an attachment) and another 19 deadlocks with either no output at all or something like this on the console: do_IRQ: 0.179 No irq handler for vector do_IRQ: 2.187 No irq handler for vector The deadlocks/panics are most definitely triggered by heavy load and past a certain point, even if the load is removed, the system will have a run away load average (20-30+) and then crash. Also worth noting is that some nodes have never experienced this mode of failure while others seem to do it every few days. Hi Joshua, thanks for data! Will take a look as only get spare time. Btw could you attach a .config file please? (if it's not a secret :) I suppose all the problematic machines has the same config? well as I see there is a series of problems. First - scheduler related deadlock by tg_share_up, and interrupt missing handlers and seems in zone memory management too :). Joshua could you turn on lock debugging at first? Created attachment 18890 [details]
2.6.27-rc8 ,config
> ------- Comment #80 from gorcunov@gmail.com 2008-11-17 00:03 -------
> well as I see there is a series of problems. First - scheduler related
> deadlock
> by tg_share_up, and interrupt missing handlers and seems in zone memory
> management too :). Joshua could you turn on lock debugging at first?
This is a chunk of the current .config:
--
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_LOCKDEP is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
--
Which lock debugging option would you like to have enabled? I believe
that enabling CONFIG_PROVE_LOCKING in the past made the problem almost
completely unreproducable but we may be fighting an entirely different
problem now.
-J
--
CONFIG_DEBUG_LOCKDEP i think. Thanks for config! At least I would ask Peter to take a look (even a glance) on tg_share_up with fool lock chain taken. IIRC there was some fixes in sched area since -rc8 but I'm not following sched code that much (unfortunatelly). Reply-To: josh@hoblitt.com > ------- Comment #83 from gorcunov@gmail.com 2008-11-17 12:04 ------- > CONFIG_DEBUG_LOCKDEP i think. Thanks for config! At least I would ask Peter > to > take a look (even a glance) on tg_share_up with fool lock chain taken. IIRC > there was some fixes in sched area since -rc8 but I'm not following sched > code > that much (unfortunatelly). I'm making a build of 2.6.28-rc5 w/ CONFIG_DEBUG_LOCKDEP to try but it may be awhile before there are any results due to the intermittancy of these crashes... Thanks, -J -- It's ok -- I think we may wait :) It appears we don't have sufficient information to find the root cause of the problem and we're not likely to collect it any time soon. Closing the bug, please reopen if necessary. |
[11532.103605] do_IRQ: 0.175 No irq handler for vector <Sep/11 12:13 pm>[11532.103613] do_IRQ: 2.175 No irq handler for vector <Sep/11 12:13 pm>[11532.103617] do_IRQ: 1.175 No irq handler for vector <Sep/11 12:14 pm>[11560.779989] do_IRQ: 0.179 No irq handler for vector <Sep/11 12:15 pm>[11622.181968] Kernel panic - not syncing: softlockup: hung tas<Sep/11 12:15 pm> <Sep/11 12:15 pm>[11622.181968] ------------[ cut here ]------------ <Sep/11 12:15 pm>[11622.181968] WARNING: at kernel/mutex.c:351 mutex_trylock+0x45/0xf6() <Sep/11 12:15 pm>[11622.181968] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Sep/11 12:15 pm>[11622.181968] Pid: 17192, comm: ppImage Not tainted 2.6.27-rc4-21704-gd25e26b #1 <Sep/11 12:15 pm>[11622.181968] <Sep/11 12:15 pm>[11622.181968] Call Trace: <Sep/11 12:15 pm>[11622.181968] <IRQ> [<ffffffff80235319>] warn_on_slowpath+0x51/0x77 <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] release_console_sem+0x3e/0x1a1 <Sep/11 12:15 pm>[11622.181968] [<ffffffff805c6031>] mutex_trylock+0x45/0xf6 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8025efc3>] crash_kexec+0x17/0xef <Sep/11 12:15 pm>[11622.181968] [<ffffffff803bb5b9>] bust_spinlocks+0x15/0x30 <Sep/11 12:15 pm>[11622.181968] [<ffffffff80235218>] panic+0x8f/0x13f <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] release_console_sem+0x3e/0x1a1 <Sep/11 12:15 pm>[11622.181968] [<ffffffff802357ce>] release_console_sem+0x3e/0x1a1 <Sep/11 12:15 pm>[11622.181968] [<ffffffff80272145>] softlockup_tick+0x19e/0x1ab <Sep/11 12:15 pm>[11622.181968] [<ffffffff8023dda4>] update_process_times+0x26/0x4b <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f7a4>] tick_periodic+0x6e/0x79 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f7c7>] tick_handle_periodic+0x18/0x59 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024f96a>] tick_do_broadcast+0x4d/0x86 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024fa20>] tick_do_periodic_broadcast+0x23/0x31 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8024fa3c>] tick_handle_periodic_broadcast+0xe/0x42 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020e9f6>] timer_event_interrupt+0x1a/0x21 <Sep/11 12:15 pm>[11622.181968] [<ffffffff80272591>] handle_IRQ_event+0x1e/0x4c <Sep/11 12:15 pm>[11622.181968] [<ffffffff80273885>] handle_edge_irq+0xe8/0x12b <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020e96f>] do_IRQ+0xf1/0x15e <Sep/11 12:15 pm>[11622.181968] [<ffffffff8020c3e1>] ret_from_intr+0x0/0xa <Sep/11 12:15 pm>[11622.181968] <EOI> [<ffffffff8021b79b>] native_flush_tlb_others+0x64/0xb3 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b7c5>] native_flush_tlb_others+0x8e/0xb3 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b7be>] native_flush_tlb_others+0x87/0xb3 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8021b8b2>] flush_tlb_page+0x5e/0x65 <Sep/11 12:15 pm>[11622.181968] [<ffffffff8022531b>] ptep_set_access_flags+0x1b/0x1f <Sep/11 12:15 pm>[11622.181968] [<ffffffff80285193>] do_wp_page+0x48b/0x51e