Bug 11629

Created attachment 17978 [details]
kernel config

Actually, on 2.6.26-rc6-00982-gb256325-dirty (last 2.6.26-rc I had) it doesn't shut down either, but it doesn't crash, it just hangs there trying to disable CPUs and they won't go down.

Reply-To: akpm@linux-foundation.org

On Tue, 23 Sep 2008 14:20:55 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=11629
> 
>            Summary: quad G5 fails to shut down
>            Product: Platform Specific/Hardware
>            Version: 2.5
>      KernelVersion: 2.6.27-rc6+wireless bits
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: PPC-64
>         AssignedTo: anton@samba.org
>         ReportedBy: johannes@sipsolutions.net
> 
> 
> Latest working kernel version: don't remember, pretty sure 2.6.26 worked but
> have been running development kernels forever
> Earliest failing kernel version: n/a, haven't checked, has been failing for a
> while
> Distribution: debian/unstable
> Hardware Environment: quad G5 powermac
> Software Environment: ?
> Problem Description: machine fails to shut down, 
> 
> Steps to reproduce:
> shut down the machine, see it fail.
> 
> http://johannes.sipsolutions.net/files/dsc_2274.jpg
> 

That went splat in the clockevents code.

I saw a pull request with several fixes float past this morning:

From: Ingo Molnar <mingo@elte.hu>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>, "H. Peter Anvin" <hpa@zytor.com>, Andrew Morton <akpm@linux-foundation.org>
Subject: [git pull] timer fixes
Date: Tue, 23 Sep 2008 21:41:00 +0200

Fingers crossed..

I'll mark this as a post-2.6.26 regression, although you seem a bit
wobbly on that.

On Tue, 2008-09-23 at 14:30 -0700, Andrew Morton wrote:

> > http://johannes.sipsolutions.net/files/dsc_2274.jpg
> > 
> 
> That went splat in the clockevents code.

Yup.

> I saw a pull request with several fixes float past this morning:
> 
> From: Ingo Molnar <mingo@elte.hu>
> To: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>, "H.
> Peter Anvin" <hpa@zytor.com>, Andrew Morton <akpm@linux-foundation.org>
> Subject: [git pull] timer fixes
> Date: Tue, 23 Sep 2008 21:41:00 +0200
> 
> Fingers crossed..

Ah, cool, I'll pull it in once I've verified this other problem I have
with ftrace on the same box.

> I'll mark this as a post-2.6.26 regression, although you seem a bit
> wobbly on that.

I was fairly sure it was ok with 2.6.26 but after checking it appears
that the crash is a post-2.6.26 regression, but the failure to shutdown
is not. Sorry, I've been following wireless-testing.git on this box for
so long it slipped my mind.

johannes

On Tue, 2008-09-23 at 14:30 -0700, Andrew Morton wrote:

> > http://johannes.sipsolutions.net/files/dsc_2274.jpg
> > 
> 
> That went splat in the clockevents code.
> 
> I saw a pull request with several fixes float past this morning:
> 
> From: Ingo Molnar <mingo@elte.hu>
> To: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>, "H.
> Peter Anvin" <hpa@zytor.com>, Andrew Morton <akpm@linux-foundation.org>
> Subject: [git pull] timer fixes
> Date: Tue, 23 Sep 2008 21:41:00 +0200
> 
> Fingers crossed..

Well, that does seem to fix the regression. That is, it no longer
crashes. It still doesn't shut down, but that seems older and possibly
partially my fault as I wrote the CPU hotplug implementation (thought it
works fine for hibernate, never mind the fact that resuming from
hibernate hangs)

johannes

On Wed, 2008-09-24 at 00:45 +0200, Johannes Berg wrote:

> Well, that does seem to fix the regression. That is, it no longer
> crashes.

Spoke too early, it seems that the crash isn't exactly reproducible all
the time nor happens at the same spot all the time. I'd suspect that the
timer code isn't to blame but something in the IPI code or so.

johannes

On Tue, 2008-09-23 at 14:30 -0700, Andrew Morton wrote:

> > Steps to reproduce:
> > shut down the machine, see it fail.
> > 
> > http://johannes.sipsolutions.net/files/dsc_2274.jpg
> > 
> 
> That went splat in the clockevents code.

This is still strange, it appears that when the CPUs fail to offline, we
still call into the clockevents code and it rightfully crashes, but I'll
check that out separately.

I've found the problem, below is a tentative fix. The problem has been
around for quite a while, here's the whole commit that "caused" it
(because it's so short):

commit 4047727e5ae33f9b8d2b7766d1994ea6e5ec2991
Author: Mark Lord <lkml@rtr.ca>
Date:   Mon Oct 1 01:20:10 2007 -0700

    Fix SMP poweroff hangs
    
    We need to disable all CPUs other than the boot CPU (usually 0) before
    attempting to power-off modern SMP machines.  This fixes the
    hang-on-poweroff issue on my MythTV SMP box, and also on Thomas Gleixner's
    new toybox.
    
    Signed-off-by: Mark Lord <mlord@pobox.com>
    Acked-by: Thomas Gleixner <tglx@linutronix.de>
    Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/kernel/sys.c b/kernel/sys.c
index 1b33b05..8ae2e63 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -32,6 +32,7 @@
 #include <linux/getcpu.h>
 #include <linux/task_io_accounting_ops.h>
 #include <linux/seccomp.h>
+#include <linux/cpu.h>
 
 #include <linux/compat.h>
 #include <linux/syscalls.h>
@@ -878,6 +879,7 @@ void kernel_power_off(void)
        kernel_shutdown_prepare(SYSTEM_POWER_OFF);
        if (pm_power_off_prepare)
                pm_power_off_prepare();
+       disable_nonboot_cpus();
        sysdev_shutdown();
        printk(KERN_EMERG "Power down.\n");
        machine_power_off();


And here's the fix.

From: Johannes Berg <johannes@sipsolutions.net>
Subject: powerpc: fix shutdown

I tracked down the shutdown regression to CPUs not dying
when being shut down during power-off. This turns out to
be due to the system_state being SYSTEM_POWER_OFF, which
this code doesn't take as a valid state for shutting off
CPUs in.

This has never made sense to me, but when I added hotplug
code to implement hibernate I only "made it work" and did
not question the need to check the system_state. Thomas
Gleixner helped me dig, but the only thing we found is
that it was added with the original commit that added CPU
hotplug support.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Joel Schopp <jschopp@austin.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
 arch/powerpc/kernel/idle.c |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

--- a/arch/powerpc/kernel/idle.c
+++ b/arch/powerpc/kernel/idle.c
@@ -36,9 +36,7 @@
 #ifdef CONFIG_HOTPLUG_CPU
 /* this is used for software suspend, and that shuts down
  * CPUs even while the system is still booting... */
-#define cpu_should_die()	(cpu_is_offline(smp_processor_id()) && \
-				   (system_state == SYSTEM_RUNNING     \
-				 || system_state == SYSTEM_BOOTING))
+#define cpu_should_die()	cpu_is_offline(smp_processor_id())
 #else
 #define cpu_should_die()	0
 #endif

I wrote:

> And here's the fix.

Well, that should of course remove that comment there too. New version
below.

From: Johannes Berg <johannes@sipsolutions.net>
Subject: powerpc: fix shutdown

I tracked down the shutdown regression to CPUs not dying
when being shut down during power-off. This turns out to
be due to the system_state being SYSTEM_POWER_OFF, which
this code doesn't take as a valid state for shutting off
CPUs in.

This has never made sense to me, but when I added hotplug
code to implement hibernate I only "made it work" and did
not question the need to check the system_state. Thomas
Gleixner helped me dig, but the only thing we found is
that it was added with the original commit that added CPU
hotplug support.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Joel Schopp <jschopp@austin.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
 arch/powerpc/kernel/idle.c |    6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

--- everything.orig/arch/powerpc/kernel/idle.c	2008-09-01 12:28:37.000000000 +0200
+++ everything/arch/powerpc/kernel/idle.c	2008-09-25 10:55:45.000000000 +0200
@@ -34,11 +34,7 @@
 #include <asm/smp.h>
 
 #ifdef CONFIG_HOTPLUG_CPU
-/* this is used for software suspend, and that shuts down
- * CPUs even while the system is still booting... */
-#define cpu_should_die()	(cpu_is_offline(smp_processor_id()) && \
-				   (system_state == SYSTEM_RUNNING     \
-				 || system_state == SYSTEM_BOOTING))
+#define cpu_should_die()	cpu_is_offline(smp_processor_id())
 #else
 #define cpu_should_die()	0
 #endif

Handled-By : Johannes Berg <johannes@sipsolutions.net>
Patch : http://bugzilla.kernel.org/show_bug.cgi?id=11629#c8

> From: Johannes Berg <johannes@sipsolutions.net>
> Subject: powerpc: fix shutdown
>
> I tracked down the shutdown regression to CPUs not dying
> when being shut down during power-off. This turns out to
> be due to the system_state being SYSTEM_POWER_OFF, which
> this code doesn't take as a valid state for shutting off
> CPUs in.
>
> This has never made sense to me, but when I added hotplug
> code to implement hibernate I only "made it work" and did
> not question the need to check the system_state. Thomas
> Gleixner helped me dig, but the only thing we found is
> that it was added with the original commit that added CPU
> hotplug support.
>
> Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Joel Schopp <jschopp@austin.ibm.com>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>   
Tested this patch on a Power6 partition with cpu hotplug online/offline 
and some shutdown and reboot cycles.  Didn't see any ill effects.

Acked-by: Joel Schopp <jschopp@austin.ibm.com>

Fixed by: http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=61e9916eba35dfb76d38013a5aae9a59cc50877a