Bug 9489 - 20000+ wake-ups/second in 2.6.24
20000+ wake-ups/second in 2.6.24
Status: CLOSED CODE_FIX
Product: Platform Specific/Hardware
Classification: Unclassified
Component: Other
All Linux
: P1 normal
Assigned To: platform_other
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-12-02 04:23 UTC by Rafael J. Wysocki
Modified: 2008-08-20 03:52 UTC (History)
6 users (show)

See Also:
Kernel Version: 2.6.24-rc
Tree: Mainline
Regression: ---


Attachments
Powertop output during the problem. (2.91 KB, text/plain)
2008-02-04 09:11 UTC, Mark Lord
Details
Kernel .config file from this incident (42.52 KB, text/plain)
2008-02-04 09:11 UTC, Mark Lord
Details
git bisect log (2.23 KB, text/plain)
2008-05-30 09:32 UTC, Márton Németh
Details
kernel .config on Clevo D410J (55.16 KB, text/plain)
2008-05-30 09:33 UTC, Márton Németh
Details

Description Rafael J. Wysocki 2007-12-02 04:23:54 UTC
Subject         : 20000+ wake-ups/second in 2.6.24
Submitter       : Mark Lord <lkml@rtr.ca>
References      : http://lkml.org/lkml/2007/12/1/141
Handled-By      : Arjan van de Ven <arjan@infradead.org>
Comment 1 Rafael J. Wysocki 2008-01-06 14:36:07 UTC
Not a regression from 2.6.23 (ref. http://lkml.org/lkml/2008/1/6/255).
Comment 2 Natalie Protasevich 2008-02-01 23:45:27 UTC
Mark Lord:

what's with powertop on 2.6.24 ??? It's gone from 100-200 wakeups/sec to 20000 wakeups/sec !!!!!!!


Analysis by Arjan:

.."something" is causing the system to go into a state
where the cpu throws us right out of the C-state the kernel asks for
...
what seems to happen (and this is based on seeing this on my own devel laptop, as well as several other reports; Mark is by far not the only one) is something hardware related, it's been seen on lots of different kernel versions.

It seems to mostly (but not 100%) happen with TI cardbus bridges, where for some reason, once the yenta driver is loaded (unloading it later makes no ifference), once in a while we get into a mode where the CPU always immediately goes out of the C-state again.
On a hardware level, there are only a few things that cause a CPU to exit a C-state, and one of them is a pending interrupt of some kind, which is the most likely thing going on here; some device or apic being stuck with interrupt high, but somehow the CPU isn't actually seeing the interrupt itself (or has it locked!)

dmesg + lsmod + lspci-v + PowerTop:
http://lkml.org/lkml/2007/12/2/75
Comment 3 Mark Lord 2008-02-04 09:11:00 UTC
Created attachment 14705 [details]
Powertop output during the problem.
Comment 4 Mark Lord 2008-02-04 09:11:56 UTC
Created attachment 14706 [details]
Kernel .config file from this incident
Comment 5 Mark Lord 2008-02-04 09:14:35 UTC
Mmmm... lost my text from the two attachments:  these are for the most recent incident of this problem, today.  Wakeups were ranging from 10000 to 36000 per second, with no obvious load on the system (97.6% idle).

/proc/interrupts didn't show anything unusual -- just the regular low rate of timer/disk interrupts, and a somewhat higher rate of EHCI interrupts.  Nothing over hundreds/second though.

Comment 6 Ray Lee 2008-03-03 14:27:38 UTC
I'm getting from 80,000 to 114,000 wakeups per second on 2.6.25-rc3. Never seen it before, but found this via archives.

I have a TI bridge: CardBus bridge: Texas Instruments PCIxx21/x515. 64 bit kernel, AMD laptop. Popping cardbus/pcmcia devices in and out of it though still works fine, so not sure if any interrupts are getting blocked. yenta is built into my kernel, so I can't test unloading/reloading. watching /proc/interrupts doesn't show anything enlightening, the most that was happening were timer interrupts at several hundred per second.
Comment 7 Natalie Protasevich 2008-03-03 15:19:04 UTC
The powertop suggestion was to disable in-kernel irq balancer, and from the config file I can see that it was indeed enabled. Have you tried disabling it by chance? (It is probably worthwhile to attach /proc/interrupts here for completeness anyway).
Comment 8 Ray Lee 2008-03-03 15:47:49 UTC
@Natalie: I'm assuming you're reacting to my comment? The attachments are from Mark Lord's original report, I'm afraid; I didn't want to confuse the issue by adding any.

Huh. In the process of answering your questions I just checked /proc/config.gz and CONFIG_X86_ACPI_CPUFREQ is not set, even though it is in my .config in my build directory. How bizarre. That can't be helping the situation.

So, please ignore this report, and I'll let you get back to your regularly scheduled bug triaging.

Sorry for the noise.
Comment 9 Natalie Protasevich 2008-03-03 16:01:27 UTC
Either that or... maybe number of wakeups are not related to the C-states, so I guess your report can be relevant indeed.
I am not sure if there is a consensus about the root cause of this yet (at least no one add to the bugzilla).
Ray, can you please clarify whan you say from 80,000 to 114,000 what was the kernel level that you had lower number of wakeups?
Comment 10 Ray Lee 2008-03-03 16:14:40 UTC
Sorry, I was ambiguous. On 2.6.25-rc3, running powertop, repeatedly hitting (r) for refresh, and the wakeups show as anywhere from (roughly) 80,000 to 114,000. All on the same kernel.

If we go with one of the theories above (or perhaps it was in the mailing list archives?) it's correlated with the CPU trying to go to sleep, so if the load increases on the system the number of wakeups from idle should go down. And (goes and compiles a new kernel) that seems to match what I'm seeing.

On previous kernels (2.6.25-rc1 and earlier), I never saw this. However, also on all previous kernels, I had CPUFREQ enabled, so *shrug*. I'm recompiling now, but will keep a watch to see if it reoccurs.
Comment 11 Mark Lord 2008-03-03 16:21:47 UTC
Nothing weird showing in /proc/interrupts .. just a few hundred per second there, despite 30000+ wakeups/sec in powertop.

I've now rebuilt kernel without *NOHZ*, to see if this improves things.

-ml
Comment 12 Ray Lee 2008-03-03 16:57:41 UTC
(In reply to comment #10)
> On previous kernels (2.6.25-rc1 and earlier), I never saw this. However, also
> on all previous kernels, I had CPUFREQ enabled, so *shrug*. I'm recompiling
> now, but will keep a watch to see if it reoccurs.

Okay, recompiled with CPUFREQ and rebooted, and it's still happening, so while this still may be user error, it's not a completely idiotic one. I'll keep investigating.
Comment 13 Márton Németh 2008-05-30 09:32:16 UTC
Created attachment 16337 [details]
git bisect log

I have similar problem on my Clevo D4J, model D410J. CONFIG_NO_HZ=y, and I get ~60 wakeups on 2.6.25, but as of 2.6.26-rc3 I get about ~90.000.

I also "git bisected" the problem, the result is:

6e908947b4995bc0e551a8257c586d5c3e428201 is first bad commit
commit 6e908947b4995bc0e551a8257c586d5c3e428201
Author: Ingo Molnar <mingo@elte.hu>
Date:   Fri Mar 21 14:32:36 2008 +0100

    x86: fix ioapic bug again
    
    un-revert:
    
     commit 4960c9df1407c2723459c60ff13e6efe0c209c62
     Author: Thomas Gleixner <tglx@linutronix.de>
     Date:   Tue Jan 22 10:23:01 2008 +0100
    
         Revert "x86: fix NMI watchdog & 'stopped time' problem"
    
             This reverts commit d4d25deca49ec2527a634557bf5a6cf449f85deb.
    
    needs a proper fix though ...
    
    Signed-off-by: Ingo Molnar <mingo@elte.hu>

:040000 040000 02918f7e330d004e1dd4c6fbb340ba4e353916df 7f9969e324e5982ebf232bb9e5f5c27f208f977b M      arch
Comment 14 Márton Németh 2008-05-30 09:33:42 UTC
Created attachment 16338 [details]
kernel .config on Clevo D410J
Comment 15 Márton Németh 2008-05-30 21:28:30 UTC
Maybe this bug related to bug #9791?
Comment 16 Márton Németh 2008-06-06 00:17:47 UTC
See also bug #10864.
Comment 17 Márton Németh 2008-08-20 02:24:19 UTC
The problem patch was reverted with commit 0b6a39f7ebcb1c82587ce35b401c513eed41ac5c in 2.6.26-rc6.

Currently I am running 2.6.27-rc3 and the problem is not there, so maybe this bug can be closed.

Note You need to log in before you can comment on or make changes to this bug.