Bug 14424 - Enabled SMP causes hrtimer_start_range_ns (tick_sched_timer) wakeups
Summary: Enabled SMP causes hrtimer_start_range_ns (tick_sched_timer) wakeups
Status: RESOLVED OBSOLETE
Alias: None
Product: Platform Specific/Hardware
Classification: Unclassified
Component: i386 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: platform_i386
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-10-16 21:51 UTC by mt229
Modified: 2013-12-10 16:57 UTC (History)
18 users (show)

See Also:
Kernel Version: 2.6.32-999
Tree: Mainline
Regression: No


Attachments
kernel config, uname -a, lspci -vvv, dmesg, messages, powertop (55.56 KB, application/x-gzip)
2009-10-16 21:51 UTC, mt229
Details
Acer Aspire One n270-kernel config, uname -a, lspci -vvv, dmesg, messages, powertop, menu.lst (58.18 KB, application/gzip)
2009-10-18 04:07 UTC, Luis Aranguren
Details
bug_14424kernel config (Apple MBP 5,5) , dmesg, lspci -vvv, powertop -d, uname -a (44.22 KB, application/x-gzip)
2009-10-23 14:21 UTC, Giorgio Gilestro
Details
config.gz for 2.6.32-rc5 (14.08 KB, application/x-gzip)
2009-10-26 16:38 UTC, Giorgio Gilestro
Details
mini.7z (36.16 KB, application/x-7z-compressed)
2010-06-20 15:32 UTC, Jayrome Noyt
Details

Description mt229 2009-10-16 21:51:11 UTC
Created attachment 23437 [details]
kernel config, uname -a, lspci -vvv, dmesg, messages, powertop

Enabled hyperthreading in BIOS causes hrtimer_start_range_ns wakeups. (~80 per second)

Reproducible: disable HT in BIOS: hrtimer_start_range_ns_wakeups (~15 per second)

Affects all kernels in Ubuntu Karmic kernel line up to 2.6.31-14-generic #47-ubuntu SMP.
Comment 1 mt229 2009-10-17 21:19:20 UTC
(In reply to comment #0)
> Created an attachment (id=23437) [details]
> kernel config, uname -a, lspci -vvv, dmesg, messages, powertop
> 
> Enabled hyperthreading in BIOS causes hrtimer_start_range_ns wakeups. (~80
> per
> second)
> 
> Reproducible: disable HT in BIOS: hrtimer_start_range_ns_wakeups (~15 per
> second)
> 
> Affects all kernels in Ubuntu Karmic kernel line up to 2.6.31-14-generic
> #47-ubuntu SMP.

Also exists with 2.6.32-999-generic #200910171000 SMP Sat Oct 17 09:56:20 UTC 2009 i686 GNU/Linux mainline vanilla kernel.
Comment 2 Luis Aranguren 2009-10-18 04:07:34 UTC
Created attachment 23457 [details]
Acer Aspire One n270-kernel config, uname -a, lspci -vvv, dmesg, messages, powertop, menu.lst

Acer Aspire One - n270 Atom

Hyperthreading is not configurable in BIOS.

Tried disabling hyperthreading in kernel options with noht which found out is just a boot and not run-time option.

Using maxcpus=1 kernel option managed to lower interrupt requests from "<kernel core> : hrtimer_start_range_ns (tick_sched_timer)" from ~60 to ~30 on idle, but with no discernible improvement in power consumption
Comment 3 mt229 2009-10-19 01:34:26 UTC
Changed the summary as it seems not to be Atom specific bug, it also is reported to occur on Dual Core:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/373245
Comment 4 Giorgio Gilestro 2009-10-23 14:21:36 UTC
Created attachment 23504 [details]
bug_14424kernel config (Apple MBP 5,5) , dmesg, lspci -vvv, powertop -d, uname -a

2.26.32-rc5, about 500 readings a second. No way for me to disable hyperthreading in bios.
Comment 5 Giorgio Gilestro 2009-10-26 16:38:01 UTC
Created attachment 23533 [details]
config.gz for 2.6.32-rc5

I am getting better result compiling the kernel with Timer Frequency at 100Hz. This was unexpected to me since my kernel is tickless anyway and I thought timer frequency doesn't matter with tickless enabled.

gg@mercurio:~$ uname -a
Linux mercurio 2.6.32-rc5 #22 SMP PREEMPT Mon Oct 26 00:14:59 CDT 2009 i686 Intel(R) Core(TM)2 Duo CPU P7550 @ 2.26GHz GenuineIntel GNU/Linux

gg@mercurio:/usr/src/linux-2.6.32-rc5$ zcat /proc/config.gz | grep HZ
CONFIG_NO_HZ=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

Powertop now says:

Top causes for wakeups:
  21.1% (110.8)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  14.8% ( 77.6)       <interrupt> : acpi 
  12.9% ( 67.6)       <interrupt> : extra timer interrupt 

This is still no optimal but is way better than before (HZ=1000) when idle kernel was giving about ~600 ticks per sec.

Kernel config.gz attached
Comment 6 Julian Sikorski 2009-10-27 16:33:48 UTC
It also happens on Core 2 Duo T7200 running an up-to-date Fedora 11:
2.6.30.8-64.fc11.x86_64
Comment 7 jm.mico 2009-11-02 19:07:53 UTC
Happens also on dell D820 running T7200 (2GHz)

2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

From powertop:
Top causes for wakeups:
  65.6% (10229.8)             smfpd : hrtimer_start_range_ns (hrtimer_wakeup)


Really annoying, is literally eating my battery!
Comment 8 jm.mico 2009-11-02 20:52:15 UTC
Interesting, the last comment I have written, was on my Kubuntu installed on my harddrive. I have checked now with an Ubuntu installed on my usb-stick and I got the following:

jmico@dell:~$ uname -a
Linux dell 2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

Top causes for wakeups:
  17.5% (  4.8)         syndaemon : hrtimer_start_range_ns (hrtimer_wakeup)
  12.4% (  3.4)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  11.7% (  3.2)       <interrupt> : ehci_hcd:usb1, uhci_hcd:usb2
  10.2% (  2.8)     <kernel core> : hrtimer_start (tick_sched_timer)
   8.8% (  2.4)    gnome-terminal : hrtimer_start_range_ns (hrtimer_wakeup)
   8.8% (  2.4)   USB device  1-4 : USB2.0 FlashDisk (Kingmax)

WIth the same kernel? How should I understand that?
Comment 9 William Lovaton 2009-11-03 03:08:26 UTC
I have the same wakeups on my system, a Fedora 12 Rawhide with the latest updates running on a Dell XPS M1210, Intel Core Duo T2400 @ 1.83GHz and kernel 2.6.31.5-96.fc12.i686.PAE.

Top causes for wakeups:
  37.6% ( 13.7)       <interrupt> : iwl3945 
  31.0% ( 11.3)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 
  10.3% (  3.8)       <interrupt> : ata_piix
   5.7% (  2.1)     <kernel core> : hrtimer_start (tick_sched_timer)

It wasn't there before on previous version of Fedora (may be 9 or 10).  What is this? is there any way to avoid those wake ups? I tried disabling the multi-core support in the BIOS and the wps decreased just slightly.
Comment 10 Thomas Gleixner 2009-11-03 10:01:07 UTC
> >From powertop:
> Top causes for wakeups:
>   65.6% (10229.8)             smfpd : hrtimer_start_range_ns (hrtimer_wakeup)

That's not a kernel problem. smfpd is a binary only user space
daemon. We can do nothing about that.

Thanks,

	tglx
Comment 11 Thomas Gleixner 2009-11-03 10:02:32 UTC
> Top causes for wakeups:
>   17.5% (  4.8)         syndaemon : hrtimer_start_range_ns (hrtimer_wakeup)
>   12.4% (  3.4)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
>   11.7% (  3.2)       <interrupt> : ehci_hcd:usb1, uhci_hcd:usb2
>   10.2% (  2.8)     <kernel core> : hrtimer_start (tick_sched_timer)
>    8.8% (  2.4)    gnome-terminal : hrtimer_start_range_ns (hrtimer_wakeup)
>    8.8% (  2.4)   USB device  1-4 : USB2.0 FlashDisk (Kingmax)
> 
> WIth the same kernel? How should I understand that?

Different user space. The USB stick does not have the Samsung binary
blob installed.

Thanks,

	tglx
Comment 12 jfm3 2009-11-24 23:14:33 UTC
Hp MINI, atom n270.

powertop also complains that the USB device is constantly in use when I have an SD card inserted.  I only mention this on the odd chance it's related.
Comment 13 fatgerman 2010-01-25 22:28:22 UTC
Just adding my vote for this bug, getting the same behaviour here on a Core II Duo laptop. Battery life is around 45 minutes. Under Windows it is > 3hrs. With kernel 2.6.26 (the last version I used before 2.6.31) it was around 2.5 hrs. This is definitely a regression.

Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 8.4%)         2.01 Ghz     0.0%
C0                0.0ms ( 0.0%)         2.00 Ghz     0.0%
C1 mwait          0.0ms ( 0.0%)         1.60 Ghz     0.0%
C2 mwait          0.0ms ( 0.0%)         1200 Mhz     0.0%
C3 mwait         10.2ms (91.6%)          800 Mhz   100.0%

Wakeups-from-idle per second : 90.4     interval: 5.0s
Power usage (ACPI estimate): 16.3W (3.4 hours)

Top causes for wakeups:
  40.1% ( 79.8)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  17.5% ( 34.8)      <kernel IPI> : Rescheduling interrupts
   9.2% ( 18.2)     <kernel core> : hrtimer_start (tick_sched_timer)

N.B. The 3.4 hour estimate from powertop is complete nonsense - I timed the time from a full battery to a battery at 5% charge at pretty close to 45 minutes.
Comment 14 fatgerman 2010-01-25 23:47:32 UTC
On further inspection, I think the hrtimer wakeups are a red herring as far as power usage is concerned.

I built a new kernel with Hi-Res Timers disabled. Battery life was about the same, even though the hrtimer_start wakeups were no longer showing. However powertop was now showing a few userspace applications as causing lots of wakeups.

So, rebooted back into my stock Ubuntu 2.6.31 kernel. Powertop is back showing the hrtimer wakeups. I then quit two of the applications that had shown up before (in my case Kopete and Klipper). The hrtimer wakeups disappeared from the list completely, and the wakeups-per-second was the lowest I have ever seen. Whether this affects battery life I have yet to investigate because it's late and the beer has run out.

I think perhaps we have some bad applications using a lot of timer wakeups and this is making it look like a kernel problem.
Comment 15 Artem S. Tashkinov 2010-02-25 11:17:18 UTC
I have a similar problem, see bug 15364.

I have Intel Core i5 650 CPU with HT enabled.
Comment 16 Dominik Brodowski 2010-03-30 14:10:40 UTC
On an Intel(R) Core(TM)2 Duo CPU     T7250  @ 2.00GHz, booting into most recent mainline (-rc2-git4) with init=/bin/bash and nohz_mode=2 leads to about 10 excessive wakeups per second, for which I have no explanation for.

powertop run #1: clocksource=hpet
  45.0% (  9.4)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
   6.1% (  1.3)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 

run #2: clocksource=hpet
  40.6% (  5.9)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
  11.9% (  1.7)       <interrupt> : extra timer interrupt 
   7.3% (  1.1)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer) 

run #3: clocksource=acpi_pm
  32.7% (  5.6)     <kernel core> : hrtimer_start (tick_sched_timer) 
...
  29.2% (  5.0)       <interrupt> : extra timer interrupt 
   5.8% (  1.0)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)

On a normal boot, while entering this bug report, I get:

  64,1% (362,2)     <Kernel Kern> : hrtimer_start_range_ns (tick_sched_timer) 
   7,2% ( 40,5)       <interrupt> : extra timer interrupt 
...
   1,5% (  8,3)     <Kernel Kern> : hrtimer_start (tick_sched_timer)
Comment 17 GNUtoo 2010-04-18 16:26:21 UTC
I think I have the same problem under ARM,which prevent the device I use from suspending.
echo userspace > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor && sleep 2 && dmesg -c > /home/root/before ;echo 1 > /proc/timer_stats ;sleep .1; echo mem > /sys/power/state ;sleep .1; echo 0 > /proc/timer_stats
produces something like that
$ cat /proc/timer_stats 
Timer Stats Version: v0.2
Sample period: 0.248 s
    1,   753 sleep            hrtimer_start_range_ns (hrtimer_wakeup)
    5,     0 swapper          hrtimer_start_range_ns (tick_sched_timer)
    1,   754 sleep            hrtimer_start_range_ns (hrtimer_wakeup)
7 total events

Unfortunately,even if the kernel I run is not tainted,it's definitely not vanilla as it was derived from the 2.6.32 android msm kernel.

I added this comment just to show that it's not bound to x86.

Denis.
Comment 18 GNUtoo 2010-04-18 16:29:08 UTC
(In reply to comment #17)

I forgot to tell our kernel is derived from 2.6.32
Comment 19 GNUtoo 2010-04-25 20:39:58 UTC
we solved our maybe unrelated issue.
http://gitorious.org/htc-msm-2-6-32/leviathan-incoming/commit/636fa4daf243826ef6cebb64dd0509f3b079fcb5

if it was not related,sorry for the inconvenience.
Comment 20 Jayrome Noyt 2010-06-20 15:32:03 UTC
Created attachment 26867 [details]
mini.7z

kernel-2.6.32.14-127.fc12.i686

on Dell mini 10v
Comment 21 Alan 2012-06-13 20:22:47 UTC
Is this still seen with modern kernels/distros ?
Comment 22 Jonathan 2012-06-19 22:34:05 UTC
It doesn't seem to be reported by powertop in recent kernel versions, but powertop has also changed a certain amount in this time.

Note You need to log in before you can comment on or make changes to this bug.