Subject : 20000+ wake-ups/second in 2.6.24 Submitter : Mark Lord <lkml@rtr.ca> References : http://lkml.org/lkml/2007/12/1/141 Handled-By : Arjan van de Ven <arjan@infradead.org>
Not a regression from 2.6.23 (ref. http://lkml.org/lkml/2008/1/6/255).
Mark Lord: what's with powertop on 2.6.24 ??? It's gone from 100-200 wakeups/sec to 20000 wakeups/sec !!!!!!! Analysis by Arjan: .."something" is causing the system to go into a state where the cpu throws us right out of the C-state the kernel asks for ... what seems to happen (and this is based on seeing this on my own devel laptop, as well as several other reports; Mark is by far not the only one) is something hardware related, it's been seen on lots of different kernel versions. It seems to mostly (but not 100%) happen with TI cardbus bridges, where for some reason, once the yenta driver is loaded (unloading it later makes no ifference), once in a while we get into a mode where the CPU always immediately goes out of the C-state again. On a hardware level, there are only a few things that cause a CPU to exit a C-state, and one of them is a pending interrupt of some kind, which is the most likely thing going on here; some device or apic being stuck with interrupt high, but somehow the CPU isn't actually seeing the interrupt itself (or has it locked!) dmesg + lsmod + lspci-v + PowerTop: http://lkml.org/lkml/2007/12/2/75
Created attachment 14705 [details] Powertop output during the problem.
Created attachment 14706 [details] Kernel .config file from this incident
Mmmm... lost my text from the two attachments: these are for the most recent incident of this problem, today. Wakeups were ranging from 10000 to 36000 per second, with no obvious load on the system (97.6% idle). /proc/interrupts didn't show anything unusual -- just the regular low rate of timer/disk interrupts, and a somewhat higher rate of EHCI interrupts. Nothing over hundreds/second though.
I'm getting from 80,000 to 114,000 wakeups per second on 2.6.25-rc3. Never seen it before, but found this via archives. I have a TI bridge: CardBus bridge: Texas Instruments PCIxx21/x515. 64 bit kernel, AMD laptop. Popping cardbus/pcmcia devices in and out of it though still works fine, so not sure if any interrupts are getting blocked. yenta is built into my kernel, so I can't test unloading/reloading. watching /proc/interrupts doesn't show anything enlightening, the most that was happening were timer interrupts at several hundred per second.
The powertop suggestion was to disable in-kernel irq balancer, and from the config file I can see that it was indeed enabled. Have you tried disabling it by chance? (It is probably worthwhile to attach /proc/interrupts here for completeness anyway).
@Natalie: I'm assuming you're reacting to my comment? The attachments are from Mark Lord's original report, I'm afraid; I didn't want to confuse the issue by adding any. Huh. In the process of answering your questions I just checked /proc/config.gz and CONFIG_X86_ACPI_CPUFREQ is not set, even though it is in my .config in my build directory. How bizarre. That can't be helping the situation. So, please ignore this report, and I'll let you get back to your regularly scheduled bug triaging. Sorry for the noise.
Either that or... maybe number of wakeups are not related to the C-states, so I guess your report can be relevant indeed. I am not sure if there is a consensus about the root cause of this yet (at least no one add to the bugzilla). Ray, can you please clarify whan you say from 80,000 to 114,000 what was the kernel level that you had lower number of wakeups?
Sorry, I was ambiguous. On 2.6.25-rc3, running powertop, repeatedly hitting (r) for refresh, and the wakeups show as anywhere from (roughly) 80,000 to 114,000. All on the same kernel. If we go with one of the theories above (or perhaps it was in the mailing list archives?) it's correlated with the CPU trying to go to sleep, so if the load increases on the system the number of wakeups from idle should go down. And (goes and compiles a new kernel) that seems to match what I'm seeing. On previous kernels (2.6.25-rc1 and earlier), I never saw this. However, also on all previous kernels, I had CPUFREQ enabled, so *shrug*. I'm recompiling now, but will keep a watch to see if it reoccurs.
Nothing weird showing in /proc/interrupts .. just a few hundred per second there, despite 30000+ wakeups/sec in powertop. I've now rebuilt kernel without *NOHZ*, to see if this improves things. -ml
(In reply to comment #10) > On previous kernels (2.6.25-rc1 and earlier), I never saw this. However, also > on all previous kernels, I had CPUFREQ enabled, so *shrug*. I'm recompiling > now, but will keep a watch to see if it reoccurs. Okay, recompiled with CPUFREQ and rebooted, and it's still happening, so while this still may be user error, it's not a completely idiotic one. I'll keep investigating.
Created attachment 16337 [details] git bisect log I have similar problem on my Clevo D4J, model D410J. CONFIG_NO_HZ=y, and I get ~60 wakeups on 2.6.25, but as of 2.6.26-rc3 I get about ~90.000. I also "git bisected" the problem, the result is: 6e908947b4995bc0e551a8257c586d5c3e428201 is first bad commit commit 6e908947b4995bc0e551a8257c586d5c3e428201 Author: Ingo Molnar <mingo@elte.hu> Date: Fri Mar 21 14:32:36 2008 +0100 x86: fix ioapic bug again un-revert: commit 4960c9df1407c2723459c60ff13e6efe0c209c62 Author: Thomas Gleixner <tglx@linutronix.de> Date: Tue Jan 22 10:23:01 2008 +0100 Revert "x86: fix NMI watchdog & 'stopped time' problem" This reverts commit d4d25deca49ec2527a634557bf5a6cf449f85deb. needs a proper fix though ... Signed-off-by: Ingo Molnar <mingo@elte.hu> :040000 040000 02918f7e330d004e1dd4c6fbb340ba4e353916df 7f9969e324e5982ebf232bb9e5f5c27f208f977b M arch
Created attachment 16338 [details] kernel .config on Clevo D410J
Maybe this bug related to bug #9791?
See also bug #10864.
The problem patch was reverted with commit 0b6a39f7ebcb1c82587ce35b401c513eed41ac5c in 2.6.26-rc6. Currently I am running 2.6.27-rc3 and the problem is not there, so maybe this bug can be closed.