Bug 14424

(In reply to comment #0)
> Created an attachment (id=23437) [details]
> kernel config, uname -a, lspci -vvv, dmesg, messages, powertop
> 
> Enabled hyperthreading in BIOS causes hrtimer_start_range_ns wakeups. (~80
> per
> second)
> 
> Reproducible: disable HT in BIOS: hrtimer_start_range_ns_wakeups (~15 per
> second)
> 
> Affects all kernels in Ubuntu Karmic kernel line up to 2.6.31-14-generic
> #47-ubuntu SMP.

Also exists with 2.6.32-999-generic #200910171000 SMP Sat Oct 17 09:56:20 UTC 2009 i686 GNU/Linux mainline vanilla kernel.

Created attachment 23457 [details]
Acer Aspire One n270-kernel config, uname -a, lspci -vvv, dmesg, messages, powertop, menu.lst

Acer Aspire One - n270 Atom

Hyperthreading is not configurable in BIOS.

Tried disabling hyperthreading in kernel options with noht which found out is just a boot and not run-time option.

Using maxcpus=1 kernel option managed to lower interrupt requests from "<kernel core> : hrtimer_start_range_ns (tick_sched_timer)" from ~60 to ~30 on idle, but with no discernible improvement in power consumption

Changed the summary as it seems not to be Atom specific bug, it also is reported to occur on Dual Core:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/373245

Created attachment 23504 [details]
bug_14424kernel config (Apple MBP 5,5) , dmesg, lspci -vvv, powertop -d, uname -a

2.26.32-rc5, about 500 readings a second. No way for me to disable hyperthreading in bios.

Created attachment 23533 [details]
config.gz for 2.6.32-rc5

I am getting better result compiling the kernel with Timer Frequency at 100Hz. This was unexpected to me since my kernel is tickless anyway and I thought timer frequency doesn't matter with tickless enabled.

gg@mercurio:~$ uname -a
Linux mercurio 2.6.32-rc5 #22 SMP PREEMPT Mon Oct 26 00:14:59 CDT 2009 i686 Intel(R) Core(TM)2 Duo CPU P7550 @ 2.26GHz GenuineIntel GNU/Linux

gg@mercurio:/usr/src/linux-2.6.32-rc5$ zcat /proc/config.gz | grep HZ
CONFIG_NO_HZ=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

Powertop now says:

Top causes for wakeups:
  21.1% (110.8)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  14.8% ( 77.6)       <interrupt> : acpi 
  12.9% ( 67.6)       <interrupt> : extra timer interrupt 

This is still no optimal but is way better than before (HZ=1000) when idle kernel was giving about ~600 ticks per sec.

Kernel config.gz attached

It also happens on Core 2 Duo T7200 running an up-to-date Fedora 11:
2.6.30.8-64.fc11.x86_64

Happens also on dell D820 running T7200 (2GHz)

2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

From powertop:
Top causes for wakeups:
  65.6% (10229.8)             smfpd : hrtimer_start_range_ns (hrtimer_wakeup)


Really annoying, is literally eating my battery!

Interesting, the last comment I have written, was on my Kubuntu installed on my harddrive. I have checked now with an Ubuntu installed on my usb-stick and I got the following:

jmico@dell:~$ uname -a
Linux dell 2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

Top causes for wakeups:
  17.5% (  4.8)         syndaemon : hrtimer_start_range_ns (hrtimer_wakeup)
  12.4% (  3.4)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  11.7% (  3.2)       <interrupt> : ehci_hcd:usb1, uhci_hcd:usb2
  10.2% (  2.8)     <kernel core> : hrtimer_start (tick_sched_timer)
   8.8% (  2.4)    gnome-terminal : hrtimer_start_range_ns (hrtimer_wakeup)
   8.8% (  2.4)   USB device  1-4 : USB2.0 FlashDisk (Kingmax)

WIth the same kernel? How should I understand that?

I have the same wakeups on my system, a Fedora 12 Rawhide with the latest updates running on a Dell XPS M1210, Intel Core Duo T2400 @ 1.83GHz and kernel 2.6.31.5-96.fc12.i686.PAE.

Top causes for wakeups:
  37.6% ( 13.7)       <interrupt> : iwl3945 
  31.0% ( 11.3)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  10.3% (  3.8)       <interrupt> : ata_piix
   5.7% (  2.1)     <kernel core> : hrtimer_start (tick_sched_timer)

It wasn't there before on previous version of Fedora (may be 9 or 10).  What is this? is there any way to avoid those wake ups? I tried disabling the multi-core support in the BIOS and the wps decreased just slightly.

> >From powertop:
> Top causes for wakeups:
>   65.6% (10229.8)             smfpd : hrtimer_start_range_ns (hrtimer_wakeup)

That's not a kernel problem. smfpd is a binary only user space
daemon. We can do nothing about that.

Thanks,

	tglx

> Top causes for wakeups:
>   17.5% (  4.8)         syndaemon : hrtimer_start_range_ns (hrtimer_wakeup)
>   12.4% (  3.4)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
>   11.7% (  3.2)       <interrupt> : ehci_hcd:usb1, uhci_hcd:usb2
>   10.2% (  2.8)     <kernel core> : hrtimer_start (tick_sched_timer)
>    8.8% (  2.4)    gnome-terminal : hrtimer_start_range_ns (hrtimer_wakeup)
>    8.8% (  2.4)   USB device  1-4 : USB2.0 FlashDisk (Kingmax)
> 
> WIth the same kernel? How should I understand that?

Different user space. The USB stick does not have the Samsung binary
blob installed.

Thanks,

	tglx

Hp MINI, atom n270.

powertop also complains that the USB device is constantly in use when I have an SD card inserted.  I only mention this on the odd chance it's related.

Just adding my vote for this bug, getting the same behaviour here on a Core II Duo laptop. Battery life is around 45 minutes. Under Windows it is > 3hrs. With kernel 2.6.26 (the last version I used before 2.6.31) it was around 2.5 hrs. This is definitely a regression.

Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 8.4%)         2.01 Ghz     0.0%
C0                0.0ms ( 0.0%)         2.00 Ghz     0.0%
C1 mwait          0.0ms ( 0.0%)         1.60 Ghz     0.0%
C2 mwait          0.0ms ( 0.0%)         1200 Mhz     0.0%
C3 mwait         10.2ms (91.6%)          800 Mhz   100.0%

Wakeups-from-idle per second : 90.4     interval: 5.0s
Power usage (ACPI estimate): 16.3W (3.4 hours)

Top causes for wakeups:
  40.1% ( 79.8)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  17.5% ( 34.8)      <kernel IPI> : Rescheduling interrupts
   9.2% ( 18.2)     <kernel core> : hrtimer_start (tick_sched_timer)

N.B. The 3.4 hour estimate from powertop is complete nonsense - I timed the time from a full battery to a battery at 5% charge at pretty close to 45 minutes.

On further inspection, I think the hrtimer wakeups are a red herring as far as power usage is concerned.

I built a new kernel with Hi-Res Timers disabled. Battery life was about the same, even though the hrtimer_start wakeups were no longer showing. However powertop was now showing a few userspace applications as causing lots of wakeups.

So, rebooted back into my stock Ubuntu 2.6.31 kernel. Powertop is back showing the hrtimer wakeups. I then quit two of the applications that had shown up before (in my case Kopete and Klipper). The hrtimer wakeups disappeared from the list completely, and the wakeups-per-second was the lowest I have ever seen. Whether this affects battery life I have yet to investigate because it's late and the beer has run out.

I think perhaps we have some bad applications using a lot of timer wakeups and this is making it look like a kernel problem.

I have a similar problem, see bug 15364.

I have Intel Core i5 650 CPU with HT enabled.

On an Intel(R) Core(TM)2 Duo CPU     T7250  @ 2.00GHz, booting into most recent mainline (-rc2-git4) with init=/bin/bash and nohz_mode=2 leads to about 10 excessive wakeups per second, for which I have no explanation for.

powertop run #1: clocksource=hpet
  45.0% (  9.4)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
   6.1% (  1.3)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 

run #2: clocksource=hpet
  40.6% (  5.9)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
  11.9% (  1.7)       <interrupt> : extra timer interrupt 
   7.3% (  1.1)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 

run #3: clocksource=acpi_pm
  32.7% (  5.6)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
  29.2% (  5.0)       <interrupt> : extra timer interrupt 
   5.8% (  1.0)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)

On a normal boot, while entering this bug report, I get:

  64,1% (362,2)     <Kernel Kern> : hrtimer_start_range_ns (tick_sched_timer) 
   7,2% ( 40,5)       <interrupt> : extra timer interrupt 
...
   1,5% (  8,3)     <Kernel Kern> : hrtimer_start (tick_sched_timer)

I think I have the same problem under ARM,which prevent the device I use from suspending.
echo userspace > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor && sleep 2 && dmesg -c > /home/root/before ;echo 1 > /proc/timer_stats ;sleep .1; echo mem > /sys/power/state ;sleep .1; echo 0 > /proc/timer_stats
produces something like that
$ cat /proc/timer_stats 
Timer Stats Version: v0.2
Sample period: 0.248 s
    1,   753 sleep            hrtimer_start_range_ns (hrtimer_wakeup)
    5,     0 swapper          hrtimer_start_range_ns (tick_sched_timer)
    1,   754 sleep            hrtimer_start_range_ns (hrtimer_wakeup)
7 total events

Unfortunately,even if the kernel I run is not tainted,it's definitely not vanilla as it was derived from the 2.6.32 android msm kernel.

I added this comment just to show that it's not bound to x86.

Denis.

(In reply to comment #17)

I forgot to tell our kernel is derived from 2.6.32

we solved our maybe unrelated issue.
http://gitorious.org/htc-msm-2-6-32/leviathan-incoming/commit/636fa4daf243826ef6cebb64dd0509f3b079fcb5

if it was not related,sorry for the inconvenience.

Created attachment 26867 [details]
mini.7z

kernel-2.6.32.14-127.fc12.i686

on Dell mini 10v

Is this still seen with modern kernels/distros ?

It doesn't seem to be reported by powertop in recent kernel versions, but powertop has also changed a certain amount in this time.