Kernel 2.6.33.5 is also unaffected. The 2.6.33.5 kernel is also tickless, I'll try to bisect tomorrow.

Here's the bisect result. Looks like it's not related to the timers at all:

[root@intranator linux-2.6]# git bisect good
ae74e823cb7d4cd476f623fce9a38f625f6c09a8 is the first bad commit
commit ae74e823cb7d4cd476f623fce9a38f625f6c09a8
Author: Martin Wilck <martin.wilck@ts.fujitsu.com>
Date:   Wed Mar 10 15:23:06 2010 -0800

    ipmi: add parameter to limit CPU usage in kipmid

    In some cases kipmid can use a lot of CPU.  This adds a way to tune the
    CPU used by kipmid to help in those cases.  By setting kipmid_max_busy_us
    to a value between 100 and 500, it is possible to bring down kipmid CPU
    load to practically 0 without loosing too much ipmi throughput
    performance.  Not setting the value, or setting the value to zero,
    operation is unaffected.

    Signed-off-by: Martin Wilck <martin.wilck@ts.fujitsu.com>
    Signed-off-by: Corey Minyard <cminyard@mvista.com>
    Cc: Jean Delvare <jdelvare@suse.de>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

:040000 040000 dde007c89f875dd7553ea13b644da5148d2b9299 b036bc77d136e9cbb702b38dd9ab6cd2af361ea5 M      Documentation
:040000 040000 1ef18e58aca5e025d0ea0691b1823ff9efd06c54 d3025083ab5f79e10ba4651815dda8d86e021766 M      drivers


I've manually checked out this commit and the commit before it
just to verify I didn't mess up the bisect process.

powertop output without the patch:
Top causes for wakeups:
  39.3% (661.0)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  14.7% (246.7)     <kernel core> : add_timer (smi_timeout) 
  14.4% (241.7)            kipmi0 : schedule_timeout_interruptible (process_timeout) 
  11.9% (200.3)     <kernel core> : hrtimer_start (tick_sched_timer) 
   9.4% (157.7)       <interrupt> : extra timer interrupt 
   5.0% ( 83.7)             squid : sys_epoll_wait (process_timeout) 
   2.6% ( 44.0)       <interrupt> : cciss0 
   0.6% ( 10.0)     <kernel core> : ipmi_timeout (ipmi_timeout)

powertop with the ipmi patch:
Top causes for wakeups:
  43.7% (858.7)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  25.3% (498.0)            kipmi0 : schedule_timeout_interruptible (process_timeout) 
  12.7% (250.0)     <kernel core> : add_timer (smi_timeout) 
   6.7% (132.0)     <kernel core> : hrtimer_start (tick_sched_timer) 
   4.2% ( 83.0)             squid : sys_epoll_wait (process_timeout) 
   2.9% ( 56.7)       <interrupt> : extra timer interrupt 
   2.1% ( 41.7)       <interrupt> : cciss0 
   0.5% ( 10.0)     <kernel core> : ipmi_timeout (ipmi_timeout)

I can sometimes even see 800 wake ups from "kipmi0" with the patch.

The sysfs text mentions it hogs the CPU by default. Shouldn't this 
be the other way around? I don't use any impitool userspace program at all.

According to the ubuntu bug report
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/575392),
this also happens on a MacBook box, I guess mobile users are
not pleased by the shortened battery life ;)

Cheers,
Thomas

On Mon, 7 Jun 2010 15:17:23 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=16147

(ae74e823cb7d4cd476f623fce9a38f625f6c09a8 "ipmi: add parameter to limit
CPU usage in kipmid" causes ksoftirq meltdown).

OK, this is tricky.  It's a 2.6.33 -> 2.6.34 regression caused by an
IPMI change.  The patch doesn't revert easily from mainline because
it's somewhat tangled up with Matthew's
3326f4f2276791561af1fd5f2020be0186459813 ("ipmi: reduce polling").

For -stable we could just revert
ae74e823cb7d4cd476f623fce9a38f625f6c09a8 as Thomas has done.  For
mainline we'd need to disentangle the revert from Matthew's change, or
revert both.  Or, of course, just fix the bug.

Corey, do you have the time to nail this issue for us?

Thanks.

I'll work on this tomorrow morning.

It must be this part of the changeset.

                        schedule();
                else
-                       schedule_timeout_interruptible(1);
+                       schedule_timeout_interruptible(0);
        }
        return 0;

Looks as if we finally found out why the old code had schedule_timeout_interruptible(1)...

First-Bad-Commit : ae74e823cb7d4cd476f623fce9a38f625f6c09a8

This is working fine on my system, with or without the change Martin suggests.

Thomas, you can try Martin's suggestion?  Looking at the patch, that's the only thing that should really change any behavior by default.  Due to some other changes, it will look like:

Index: linux-2.6/drivers/char/ipmi/ipmi_si_intf.c
===================================================================
--- linux-2.6.orig/drivers/char/ipmi/ipmi_si_intf.c
+++ linux-2.6/drivers/char/ipmi/ipmi_si_intf.c
@@ -1018,7 +1018,7 @@ static int ipmi_thread(void *data)
 		else if (smi_result == SI_SM_IDLE)
 			schedule_timeout_interruptible(100);
 		else
-			schedule_timeout_interruptible(0);
+			schedule_timeout_interruptible(1);
 	}
 	return 0;
 }

Created attachment 26732 [details]
revert default behavior back to schedule_timeout_interruptible(1)

Patch works like a charm!

Powertop output with the patch applied:
Top causes for wakeups:
  57.1% (783.0)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  18.2% (250.0)            kipmi0 : schedule_timeout_interruptible (process_timeout) 
   9.1% (125.0)     <kernel core> : add_timer (smi_timeout) 
   6.1% ( 83.3)             squid : sys_epoll_wait (process_timeout) 
   5.7% ( 78.3)       <interrupt> : extra timer interrupt 
   0.7% ( 10.0)     <kernel core> : ipmi_timeout (ipmi_timeout) 

The box is idle again. Thanks.

Handled-By : Martin Wilck <martin.wilck@ts.fujitsu.com>
Patch : https://bugzilla.kernel.org/attachment.cgi?id=26732

Fixed by commit 8d1f66dc9b4f80a1441bc1c33efa98aca99e8813 .