Bug 3341 - Kernel clock is running 3 times too fast
Summary: Kernel clock is running 3 times too fast
Status: CLOSED PATCH_ALREADY_AVAILABLE
Alias: None
Product: Timers
Classification: Unclassified
Component: Interval Timers (show other bugs)
Hardware: i386 Linux
: P2 high
Assignee: Andi Kleen
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-09-04 10:09 UTC by fastclock
Modified: 2009-10-02 16:38 UTC (History)
11 users (show)

See Also:
Kernel Version: 2.6.8-1.521
Subsystem:
Regression: ---
Bisected commit-id:


Attachments
dmesg (17.09 KB, text/plain)
2005-08-02 13:54 UTC, Olivier Fourdan
Details
lspci -vv (12.52 KB, text/plain)
2005-08-02 13:55 UTC, Olivier Fourdan
Details
/proc/interrupts (650 bytes, text/plain)
2005-08-02 13:56 UTC, Olivier Fourdan
Details
dmidecode (7.23 KB, text/plain)
2005-08-02 13:57 UTC, Olivier Fourdan
Details
Patch for Linux kernel 2.6.12 (13.33 KB, patch)
2005-10-30 13:32 UTC, Olivier Fourdan
Details | Diff
Patch for Linux kernel 2.6.13 (13.42 KB, patch)
2005-10-30 13:33 UTC, Olivier Fourdan
Details | Diff
Patch for Linux kernel 2.6.14 (13.01 KB, patch)
2005-10-30 13:33 UTC, Olivier Fourdan
Details | Diff
Kernel log with R200 patch applied (15.03 KB, application/octet-stream)
2005-11-21 13:33 UTC, Olivier Fourdan
Details
Kernel log with R200 patch applied (new) (15.08 KB, text/plain)
2005-11-21 13:45 UTC, Olivier Fourdan
Details
Kernel log with Andi's patch applied (17.16 KB, text/plain)
2005-11-26 12:56 UTC, Olivier Fourdan
Details
Patch for kernel 2.6.12 (14.06 KB, patch)
2005-11-27 13:49 UTC, Olivier Fourdan
Details | Diff
patch for kernel 2.6.13 (13.95 KB, patch)
2005-11-27 13:50 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.14 (13.67 KB, patch)
2005-11-27 13:50 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.15-rc2 (13.49 KB, patch)
2005-11-27 13:51 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.14 (13.11 KB, patch)
2005-11-28 14:19 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.15-rc2 (12.91 KB, patch)
2005-11-28 14:20 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.15 (252 bytes, patch)
2006-01-04 13:22 UTC, Olivier Fourdan
Details | Diff
Patch for kernel 2.6.15 (12.63 KB, patch)
2006-01-06 14:23 UTC, Olivier Fourdan
Details | Diff
Calibrate APIC timer using PM timer (4.04 KB, patch)
2006-01-30 22:58 UTC, Andi Kleen
Details | Diff
Log of 2.6.16-rc1-mm4 with apic-main-timer.patch and apic-pm-timer.patch (33.51 KB, text/plain)
2006-01-31 12:52 UTC, Olivier Fourdan
Details
nx9105 dmesg and dmidecode (22.65 KB, text/plain)
2009-10-02 16:38 UTC, John Talbut
Details

Description fastclock 2004-09-04 10:09:56 UTC
Distribution: Fefora Core 2

Hardware Environment: HP Pavilion zv5000z notebook PC - AMD Athlon64 3000+, AMD
K8 Northbridge, nVidia nForce3 PCI bridge, 64 MB nVidia GeForce 4 440 Go
graphics, 768 MB PC2700 SODIMM...

Software Environment: Linux x86_64 2.6.8-1.521 (and all earlier versions
including 2.4, 32-bit and 64-bit kernels)

Problem Description:
The Linux kernel is having intermittent problems with detecting and setting up
the system clock correctly. The Linux system clock is scaled 3 times too fast
for about 9 out of every 10 cool boot sessions. Rebooting does not appear to
change the behavior of the Linux system clock. If I powered on the machine and
happened to get a normal Linux system clock, I can reboot repeatedly and the
Linux system clock would still be running fine. However, if I powered off and on
the machine, I would have about a 90% chance of getting the fast Linux clock again.

My system dual-boots Linux and Windows XP. The Windows clock is always running
fine. The hardware clock is also running fine - I can see that while the
GRUB boot loader is counting down (in normal time).  But when the Linux
kernel takes over, the system (software) clock would likely be scaled 3 times
too fast.

Followings are the differences in dmesg logs between Linux sessions with normal
and fast system clocks:
Normal > time.c: Detected 797.960 MHz processor.
Fast >   time.c: Detected 265.995 MHz processor.
...
Normal > Calibrating delay loop... 1581.05 BogoMIPS
Fast >   Calibrating delay loop... 517.12 BogoMIPS
...
Normal > Detected 12.468 MHz APIC timer.
Fast >   Detected 4.156 MHz APIC timer.

Steps to reproduce:
On a comparable PC, trying power the machine off and on several times and
observe the speed of the Linux system clock.

I can always reproduce the problem here and send dmesg logs.
Comment 1 Jeramy Rutley 2004-09-21 14:14:18 UTC
I can also reproduce this error, and have seen it in 2.6.6 and later kernels on
my HP Omnibook 6000.

I can also supply dmesg logs, if desired.
Comment 2 john stultz 2005-08-02 13:01:49 UTC
This issue seem simlar to bug #3927. 

Does booting w/ "noapic" change anything? Does booting w/ "no_timer_check" help?
Comment 3 Olivier Fourdan 2005-08-02 13:34:09 UTC
> This issue seem simlar to bug #3927. 

I don't think so. in the case of #3927, the system was showing the bug in Linux.
With this bug the system shows the bug everywhere (Linux, FreeBSD, memtest86+)
even within the BIOS startup sequence.

Plus #3927 was with ATI while #3341 is with NVidia.

> Does booting w/ "noapic" change anything? Does booting w/ "no_timer_check" help?

No that makes no difference. The "noapic" flag seems to prevent the system from
booting, but still the computed CPU speed is wrong (as mentionned 3 times lower
than its actual value).

the "no_timer_check" has no effect.

It's worth noting that the issue doesn't show on all similar models, but shows
on some, either 32bit or 64bit CPU. Even there, it doesn't show all the time,
sometimes the system works just fine.
Comment 4 Olivier Fourdan 2005-08-02 13:54:42 UTC
Created attachment 5480 [details]
dmesg

dmesg (all timings are wrong, thus the forced lpj otherwise the system can't
read the disk)
Comment 5 Olivier Fourdan 2005-08-02 13:55:16 UTC
Created attachment 5481 [details]
lspci -vv
Comment 6 Olivier Fourdan 2005-08-02 13:56:00 UTC
Created attachment 5482 [details]
/proc/interrupts
Comment 7 Olivier Fourdan 2005-08-02 13:57:38 UTC
Created attachment 5483 [details]
dmidecode

If that's on any help. What I find odd is that the SMBios gives 1.6V for the
CPU while powernow-k8 gives 1.550V max.
Comment 8 john stultz 2005-08-02 14:07:17 UTC
Disabling powernow-k8 in the kernel doesn't help either?

Andi: Do you have any clue here?
Comment 9 john stultz 2005-08-02 14:08:24 UTC
Oh, and could you please revalidate that it occurs with 2.6.13-rc5?
Comment 10 Olivier Fourdan 2005-08-02 23:05:53 UTC
> Disabling powernow-k8 in the kernel doesn't help either?
No, makes no difference. Seems powernow is loaded later in the boot process.

> Oh, and could you please revalidate that it occurs with 2.6.13-rc5?
Makes no difference either.
Comment 11 Olivier Fourdan 2005-08-03 13:38:40 UTC
Just to make 100% sure, I tried the patch
http://www.ussg.iu.edu/hypermail/linux/kernel/0504.0/1625.html but that doesn't
fix the issue.

With that patch, IRQ0 is labeled XT-PIC but the PIT still goes 3 times too fast.
Also the LOC and IRQ0 counter are in sync.
Comment 12 Olivier Fourdan 2005-08-07 12:46:35 UTC
Could it be some erratum in the southbridge (nforce3) that wouldn't be handled
by the BIOS that could explain that behaviour?

To me it doesn'look like an IRQ routing issue, but really like a rate 3x too
fast in the 82C54, any channel, any mode: if the timer goes 3 times too fast,
the following test;

    grep LOC /proc/interrupts && sleep 10 && grep LOC /proc/interrupts
    LOC:     212737
    LOC:     222772

*but* the sleep last for approx 3 secs (using the watch).

Does anybody has access to nforce3 errata (btw, I'm not even sure the 82C54 is
in the nforce3)? I could find a lot of documentation from AMD but nothing from
NVidia.
Comment 13 Olivier Fourdan 2005-08-10 12:42:57 UTC
Ok, I have some interesting findings. I've always been surprised that Windows
got the timer right while any other system or evcen application was getting the
timer 3 times too fast (see comment on LKML here http://lkml.org/lkml/2005/7/27/7)

After digging a bit, I found that the timer management in windows depends on the
HAL used. I also found that the HAL installed by default on my laptopn is "ACPI
uniprocessor".

I found on Microsoft web site a method to force a Hardware Abstraction Layer
(http://support.microsoft.com/?kbid=299340)

I therefore ripped off my Windows isntalled and used the "Stanard PC " HAL
instead. And guess what? The timre runs 3 times to fast in Windows XP too.

So, my guess is that Windows XP using the ACPI HAL does the same as my patch, ie
calibrate the PIT using the ACPI timer (I actually think Windows uses a variable
tick calibrated on the ACPI timer)

So you think my patch (www.xfce.org/~olivier/r3000) or a similar patch could be
included in the standard kernel ?
Comment 14 Olivier Fourdan 2005-08-10 12:46:34 UTC
PS: The patch on http://www.xfce.org/~olivier/r3000/ has been improved over the
fist one I posted on LKML and is functionning well on i386 and x86_64 architectures.
Comment 15 john stultz 2005-08-10 15:23:20 UTC
Olivier: The solid feedback (and patches even!) you've been providing has been
great. I'm not sure if you're patch is really the best approach. I'm hoping
there is some nforce3 errata we can find that specifies what needs to be done to
 get the timer interrupt frequency properly set on a warm reboot. That way we
can specifically blacklist the chipset and special boot options won't be necessary.

Comment 16 Olivier Fourdan 2005-08-11 03:05:00 UTC
John, just for correctness, it's not on warm reboot, it's on cold boot. If by
chance the system boots with the timer right, the timer stays right until the
computer is turned off and on again.

As for the errata I doubt there's one, because Windows would have it implemented
then (w/out ACPI enabled, the problem shows even in XP with all latest available
updates)
Comment 17 Thomas Johnson 2005-09-02 17:10:41 UTC
Just wanted to say that I was having similar problems on an Athlon64X2 machine
with 2.6.12, and the patch fixed those for me.
Comment 18 Olivier Fourdan 2005-10-14 12:14:51 UTC
John, I've updated my patch and I have it for 2.6.14-rc4. I guess there is still
no chance for inclusion in the kernel in that form or another (ie maybe part of
your own timer patches as an option or something)
Comment 19 Olivier Fourdan 2005-10-30 13:32:19 UTC
Created attachment 6420 [details]
Patch for Linux kernel 2.6.12
Comment 20 Olivier Fourdan 2005-10-30 13:33:12 UTC
Created attachment 6421 [details]
Patch for Linux kernel 2.6.13
Comment 21 Olivier Fourdan 2005-10-30 13:33:47 UTC
Created attachment 6422 [details]
Patch for Linux kernel 2.6.14
Comment 22 Olivier Fourdan 2005-10-30 13:36:03 UTC
Attaching patches directly to the bug report as the URL on my sites aren't
stable these days (sorry for the noise).

Once the patched kernel is built and installed, add the boot parameter to
instruct the kernel to "trust" the ACPI PM timer :

On i386, add :

clock=pmtmr

On x86_64, add :

pmtmr

Comment 23 Andi Kleen 2005-10-30 15:53:32 UTC
No way I'm adding the three times read hack into x86-64 pmtmr.  And there
is no evidence it is even needed.

What needs to be done instead is to find out why the TSC is miscalibrated.
 
Does it still happen on 2.6.14?
Comment 24 Olivier Fourdan 2005-10-30 23:54:35 UTC
 
> Does it still happen on 2.6.14?

Yes, that's why I've made the patch available for 2.6.14 too.
Comment 25 john stultz 2005-11-21 12:37:55 UTC
Olivier, Could you try the patch posted here to see if it helps?

http://marc.theaimsgroup.com/?l=linux-kernel&m=113249769027262&w=2
Comment 26 Olivier Fourdan 2005-11-21 12:49:54 UTC
Yeap, I've seen taht patch on the LKML.

I'll try and report back. But I would be somehow surprised if that help in my
case for the following reasons:

1) In my case, it's the i8259A rate that's broken
2) The R200 case looks different from my problem. My problem shows as early as
the BIOS at boot and show only intermitendly. I don't think the problem with ATI
is intermittent.
3) I still believe the problem here is a hardware bug that is hidden by Windows
implementation of the timer.

Anyway, I'll try it right now.
Comment 27 Olivier Fourdan 2005-11-21 13:09:08 UTC
I tried and it makes no difference here.
I have a kernel log if needed.
Comment 28 Olivier Fourdan 2005-11-21 13:15:31 UTC
Seems there are several different causes to the same symtoms (unfortunately for
me). On my system (and quite a few, but not all, of these R3000 HP laptops) I'm
pretty sure it's a hardware bug.

The only way to get around it is to "fix" the timer rate by comparing it with
another viable time source, which is just what my patch does.
Comment 29 john stultz 2005-11-21 13:19:56 UTC
Olivier: That's unfortunate, I appreciate your time in trying the patch though.
If you could, dmesg output would still be helpful. Thanks again!
Comment 30 Olivier Fourdan 2005-11-21 13:33:51 UTC
Created attachment 6639 [details]
Kernel log with R200 patch applied

The patch has been applied on both archs i386 and x86_64 on a 2.6.15-rc2
kernel.

I'm currently running in x86_64 mode.
Comment 31 Olivier Fourdan 2005-11-21 13:40:46 UTC
Comment on attachment 6639 [details]
Kernel log with R200 patch applied

Better (more complete log) available
Comment 32 Olivier Fourdan 2005-11-21 13:45:33 UTC
Created attachment 6640 [details]
Kernel log with R200 patch applied (new)

The kernel is 2.6.15-rc2 patched with the suggested patch (applied on the
x86_64 tree).

The previous log was truncated (log buffer too small on recent kernels)

This new one shows the issue at boot.

PM-Timer running at invalid rate: 33% of normal - aborting.
time.c: Using 3.579545 MHz PM timer.
time.c: Detected 731.484 MHz processor.
time.c: Using PIT/TSC based timekeeping.
Comment 33 Arkadiusz Miskiewicz 2005-11-22 02:03:56 UTC
FYI I have kernel with both patches applied (from comments #21 and #25) and my 
kernel runs twice fast on i686 smp kernel run on i686 up machine. Without these 
patches everything is ok.
Comment 34 Arkadiusz Miskiewicz 2005-11-22 02:05:08 UTC
I mean clock of course.
Comment 35 Olivier Fourdan 2005-11-26 12:48:02 UTC
Just tried Andi's patch posted on the ML http://lkml.org/lkml/2005/11/26/32

Makes no difference.
Comment 36 Olivier Fourdan 2005-11-26 12:56:54 UTC
Created attachment 6688 [details]
Kernel log with Andi's patch applied
Comment 37 Andi Kleen 2005-11-26 13:34:58 UTC
Looking at your log it's probably a different problem - not the APIC routing
issue. It might be related to the fact that your system boots with 800Mhz
and then the CPU is later scaled up.

Does it work correctly with "notsc" or not loading any powernow modules?
Comment 38 Olivier Fourdan 2005-11-26 14:16:03 UTC
Unfortunately not, the notsc has no effect neither does removing the powernow
modules.

The problem is that the interrupts run 3x too fast. The CPU is initialized at
2200MHz by the BIOS (that's sure, it was the fix of the latest F35 BIOS), but it
really looks like the hardware generates 3x too much interrupts, so the detected
(computed) cpu speed is 3x lower that it's actual value.

That has some very nasty effects too as the DMA timings are all broken (got a
bunch of fake HD errors because of that).
Comment 39 Andi Kleen 2005-11-26 15:06:01 UTC
Your bootlog says it starts with 800Mhz. With removing powernow I meant
never loading them - did you do that?

The detected CPU speed won't directly impact the interrupt rate because
it's fixed - in your case it's generated by the PIT which runs
with a defined frequency (unless the hardware is totally broken) 

Then there is the local APIC timer which is used for timing event
with APIC on , but that should also run independent
of the CPU frequency because it's clocked off the northbridge clock.
In your case it seems to be miscalibrated though. The bootlog 
says its running with 4Mhz,  but normally on a 800 HT systems it's
supported to run with ~12Mhz (at least it does that on my desktop)

The APIC clock is supposed to be constant, but perhaps it's not
the case. I suspect your problem will go away if you keep 
the system at 800Mhz. Correct?



Comment 40 Olivier Fourdan 2005-11-26 23:28:11 UTC
> Your bootlog says it starts with 800Mhz. With removing powernow I meant
> never loading them - did you do that?

Yes, sure.

> The detected CPU speed won't directly impact the interrupt rate because
> it's fixed - in your case it's generated by the PIT which runs
> with a defined frequency (unless the hardware is totally broken) 

The hardware *is totally broken*. It's what I'm trying to tell from the very
first time.

But i's not just my laptopn, it's quite a few of the same model. Not all though,
some are fine :/

> Then there is the local APIC timer which is used for timing event
> with APIC on , but that should also run independent
of>  the CPU frequency because it's clocked off the northbridge clock.
> In your case it seems to be miscalibrated though. The bootlog 
> says its running with 4Mhz,  but normally on a 800 HT systems it's
> supported to run with ~12Mhz (at least it does that on my desktop)

The CPU always runs at ~2200MHz and the APIC timer runs at ~12,455MHz . But as
the PIT timer that is used to calibrate those values run 3x too fast, the
computed values are 3x too low. Simple.

2200/3 = 733MHz which is the wrong computer speed displayed in the boot log
(actually 731Mhz)

> The APIC clock is supposed to be constant, but perhaps it's not
> the case. I suspect your problem will go away if you keep 
> the system at 800Mhz. Correct?

The problem never, ever goes away. It appears after a short power off/power on
cycle. Once it appears, I have to leave the laptop off for about an hour to get
a chance to have the system run at a correct speed.

That's why I've made my patch that computes the correct rate to pass to the PIT
timer to get the real interrupt rate.

The algorithm I use is simple:

- Program the PIT at a good value
- Wait for a short period of time
- Computer the ratio to program the PIT to get a correct interrupt rate
(Sometimes it's 1, sometimes it's 3, randomly)

Windows wit hthe ACPI HAL does not show the problem because I beleive it does
something somehow similar, it uses the PM timer to calibrate its variable tick rate.
Comment 41 Olivier Fourdan 2005-11-27 13:49:25 UTC
Created attachment 6692 [details]
Patch for kernel 2.6.12

New version of the patch for kernel 2.6.12
Comment 42 Olivier Fourdan 2005-11-27 13:50:13 UTC
Created attachment 6693 [details]
patch for kernel 2.6.13

New version of the patch for kernel 2.6.13
Comment 43 Olivier Fourdan 2005-11-27 13:50:57 UTC
Created attachment 6694 [details]
Patch for kernel 2.6.14

New version of the patch for kernel 2.6.14
Comment 44 Olivier Fourdan 2005-11-27 13:51:56 UTC
Created attachment 6695 [details]
Patch for kernel 2.6.15-rc2

New version of the patch for kernel 2.6.15-rc2
Comment 45 Olivier Fourdan 2005-11-27 13:57:18 UTC
Sorry for the mail traffic :(

I've attached new versions of my patch, that fixes the issue decribed at #33 on
i386, and also, I believe, a less intrusive change to the existing code.

The ration is used only if "pmtmr" is explicitely given on the cmd line (on
i386, add "clock=pmtmr", on x86_64, simply add "pmtmr") and also if the PM Timer
rate is out of the 5% bounds.

The patches are also available from http://www.xfce.org/~olivier/r3000/
Comment 46 Andi Kleen 2005-11-27 20:35:21 UTC
Ok I see what your patch is doing.

One might consider to just refuse to support such broken hardware
I wonder what the hardware designers were thinking? 

Anyways -  Did you really need the "read pmtimer three times" hack?
I would prefer not to add that unless absolutely needed.

And why are you adding the verify pmtmr function to 64bit when
you're saying that the PIT doesn't run at the corect frequency?
That doesn't make much sense because it cannot even work on your
system.

Also please look up the concept of include files instead of spreading
externs everywhere.
Comment 47 Andi Kleen 2005-11-27 20:43:47 UTC
And can someone add dmidecode output for that broken machine?
Comment 48 Olivier Fourdan 2005-11-27 22:16:33 UTC
Sure, it's already there:
http://bugzilla.kernel.org/attachment.cgi?id=5483&action=view
Comment 49 Olivier Fourdan 2005-11-28 11:39:01 UTC
> Ok I see what your patch is doing.

Thanks!

> One might consider to just refuse to support such broken hardware

Absolutely, there is nothing I can argue here. This hardware is clearly designed
and tested with Windows. And it works with Windows. So if more laptop builders
just test their hardware with that system, we might see more of these timer
problems in the future. Just a rough (pessimistic) guess though.

> I wonder what the hardware designers were thinking? 

Unfortunately I can't answer this question. And I wonder what I was thinking
when I bought it. But that's definitely another story... :)

> Anyways -  Did you really need the "read pmtimer three times" hack?
> I would prefer not to add that unless absolutely needed.

Possibly not. It was like that in the i386 version so I just started from that.
I can surely test without...

> And why are you adding the verify pmtmr function to 64bit when
> you're saying that the PIT doesn't run at the corect frequency?
> That doesn't make much sense because it cannot even work on your
> system.

The thing is that the problem is intermitent. Sometimes the PIT runs at the
correct frequency. The verify_pmtmr_rate() function is really the heart of the
patch. It compute the ratio to apply to the PIT timer to get the correct rate.
If the PMTMR is within the 5% limits, the ratio is simply not applied (this is
totally similar to what's done for i386).

> Also please look up the concept of include files instead of spreading
> externs everywhere.

Sure ;) The point was to keep the patch as little intrusive as possible. I was
afraid of adding or changing yet another header in the kernel.
Comment 50 Olivier Fourdan 2005-11-28 14:19:35 UTC
Created attachment 6708 [details]
Patch for kernel 2.6.14

New patch for 2.6.14 made after Andi's comments.
Comment 51 Olivier Fourdan 2005-11-28 14:20:30 UTC
Created attachment 6709 [details]
Patch for kernel 2.6.15-rc2

New patch for 2.6.15-rc2 made after Andi's comments.
Comment 52 Olivier Fourdan 2006-01-04 13:22:35 UTC
Created attachment 6933 [details]
Patch for kernel 2.6.15
Comment 53 Andi Kleen 2006-01-04 13:40:52 UTC
I decided giving up and just switching to the lapic timer on these boards.
I implemented this some time ago anyways for my noidletick patchkit,
but never pushed it because one of my test machines didn't like it
(they stopped the apic timer in C3). Just need to extract that code.

But the problem how to detect the problem and turn it on. For ATI it's easy.
For the broken nforce3 boards it's difficult. Perhaps your way of 
testing it against the pmtimer is reasonable.

P.S.: Your last patch is some html file, but it doesn't matter.
Comment 54 Olivier Fourdan 2006-01-06 14:23:02 UTC
Created attachment 6946 [details]
Patch for kernel 2.6.15

> P.S.: Your last patch is some html file, but it doesn't matter.
Woops, sorry, that should fix it.
Comment 55 Andi Kleen 2006-01-19 01:17:42 UTC
After various tries I gave up on the PIT routing on ATI.
New approach simply uses the APIC timer to drive the main clock. I had
implemented this long ago for dynticks, but didn't push it 
back then because there was one buggy HP nforce3 laptop
where the BIOS broke the APIC timer in C3. 

Another problem right now is that it doesn't work on Intel systems for unknown
reasons, but on those PIT typically works.

Anyways, people who see problems here please test
ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/x86_64-2.6.16-rc1-060118-2.bz2

Comment 56 Olivier Fourdan 2006-01-19 12:48:00 UTC
> Anyways, people who see problems here please test
> ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/x86_64-2.6.16-rc1-060118-2.bz2

Andi, I tried your patch and that doesn't fix the issue with the timer running
3x too fast.
Comment 57 Andi Kleen 2006-01-19 13:07:23 UTC
Wait, you're the person with the Nforce3 and the totally miswired PIT timer in
hardware right?  For that one it is not intended to work - the APIC timer is
still calibrated from the PIT even with that patch. To solve that one it would
need to be calibrated from the pmtimer. To be honest your bug is low 
priority for me because it seems to be just a single monkeyed up board type, not
a  systematic problem happening on a lot of machines.

I was mostly interested in feedback from ATI and Nforce4 users with too
fast timers right now.  

Unfortunately this bug is a wild mix of various different but unrelated
problems that just happen to have similar symptons. Maybe it would be 
best to fork it.
Comment 58 Olivier Fourdan 2006-01-19 13:18:54 UTC
Sorry, but the bug is exactly the problem I have with a similar HP laptop with
the same hardware (please reread the original bug description). Maybe you got
confused with anither bug report (there are wuite a few for ATI too)
Comment 59 Andi Kleen 2006-01-19 13:31:10 UTC
Hmm, yes indeed. Must have hit the wrong bug (strangely it seems to be the only
one of the t imer bugs assigned to me) Ok ignore the patch. Sorry.

But to fix your problem would just need to calibrate cpu_khz with the PMtimer
instead of the PIT I think - then the APIC timer would be calibrated to the
correct frequency and it should just wokr.

If you could convert your patch into just doing that on top of the patch i
posted earlier I would apply the patch. Thanks.

Comment 60 Olivier Fourdan 2006-01-19 13:46:22 UTC
> If you could convert your patch into just doing that on top of the patch i
> posted earlier I would apply the patch. Thanks.

Sounds interesting :) I'll try to prepare a patch then.
Comment 61 john stultz 2006-01-19 14:02:55 UTC
Andi: Well, its more then calibrating cpu_khz, as if the timer interrupts are
coming in too fast, jiffies will be increasing too quickly. Thus the timer
frequency has to be estimated and re-programmed (as dont in Oliver's patch) to
provide something close to ACT_HZ freq. 

Olivier: I appreciate your persistence in maintining your current patch. While I
still don't think it is acceptable, maybe the following ideas might allow it to
go in:

1) Could we avoid detecting this via the pmtmr calibration, and instead use a
DMI blacklist so we make sure we only trigger this on known problematic systems.
This also avoids the need for redefining existing boot time options (it would
just work :)

2) The latch adjustment should probably be cleaned up a bit. I'm not confident
you've got all the LATCH users fixed (apm suspend?) and the redudency points to
a need to clean up the PIT programming code.

3) Could you provide some clock drift numbers to show how good the re-estimated
and programmed frequency is to the expected HZ frequency?

thanks
-john
Comment 62 Andi Kleen 2006-01-19 14:05:08 UTC
The patch I posted avoids this problem completely by using 
the local APIC timer. It won't work on all platforms, but should 
on near all of the affected AMD systems.

Comment 63 john stultz 2006-01-19 14:06:41 UTC
Andi: I'll take that back upon second reading of what your patch does.
Re-calibrating the cpu_khz would probably be sufficent.

Although I would like to see the conditional for this be a machine specific
blacklist.
Comment 64 Andi Kleen 2006-01-19 14:19:12 UTC
Why? If APIC timer works it is great because it will be much friendlier
to dyntick. And it was always quite dumb to run two different timers on the BP.

And I'm frankly sick and tired of dealing with interrupt 0 routing issues all
the time.  XP apparently uses the APIC timer, so this part is very undertested
in BIOS and hardware and sometimes hopelessly broken (like in the laptop
in this bug) 

It won't work on newer Intel CPUs though and at least one broken AMD bios.
So we won't get rid of this can of worms completely, but at least it will
be better. I also need to still figure out why it doesn't work even
on older Intel CPUs.

Comment 65 john stultz 2006-01-19 14:24:54 UTC
Olivier: I realize you might not be able to freely change arch or boot just any
kernel, but if possible I'd be very interested to hear what happens on your
system w/ 2.6.16-rc1-mm1 for i386.

Andi: Not sure if our discussion is interleaving. Regarding comment #64, I'm
just suggesting the conditional for recomputing cpu_khz based on the pmtimer
instead of the PIT to be conditional on this system being broken. That way we
have some documenation on which hardware has which problems, so future changes
dont' bring out this issue agian.

For i386 we may need something like what I suggested in comment #61, or some
conditional port of your lapic usage as well.
Comment 66 Andi Kleen 2006-01-19 14:31:17 UTC
Ah ok - i thought you were arguing against using it for the main timer.
I see your point regarding the calibration. I considered that too.

If the code decides to use pmtimer it should use pmtimer based calibration.

This laptop won't though because it is single core - in this case it will
indeed need a DMI entry to force pmtimer.
Comment 67 Olivier Fourdan 2006-01-29 13:47:02 UTC
Hi John,

In reply to your comment #61:

> 1) Could we avoid detecting this via the pmtmr calibration, and instead use a
> DMI blacklist so we make sure we only trigger this on known problematic systems.
> This also avoids the need for redefining existing boot time options (it would
> just work :)

Nope, I doubt it would work (unless I misunderstood your comment) because the
problem is intermittent (as described in the bug description). It means that
sometimes the system boots right, sometimes it runs 3x too fast. So blacklisting
won't help here. Plus, most of those laptops just work fine, blacklisting them
will lead to a bunch of problems.

That's why I do believe calibration is the only way to work arround that
problem. I perfectly understand that such a fix may not be suitable in the
general case, but the proposed solution is technically the safest way to work
arround the problem. It may or may not fit, that's another problem. 

It would be just great if we could have something working as it's a big problem
using Linux on that laptop (which makes the laptop mostly useless for me without
the patch.)

Cheers
Olivier.
Comment 68 Andi Kleen 2006-01-29 20:34:09 UTC
The problem I have with calibration is that if we calibrate e.g. 
the PIT against the pmtimer how do we decide which is one is wrong and which
one is right? Apparently there are both systems with wrong PIT and wrong
pmtimer (at least on i386, haven't heard of such a 64bit beasts, but I'm
sure some motherboard designer will screw that up sooner or later) 

So it needs some table based on system and command line options.

I don't know what you mean with blacklisting will break many of these
laptops. With DMI it's possible to only exclude very specific models.

So the only patch still needed is some way to calibrate the APIC timer
against pmtimer and then the rest can be done with DMI entries/command
line options.  I thought you wanted send me a patch for that? 
If not I'll do it myself.
Comment 69 john stultz 2006-01-30 12:53:26 UTC
>1) Could we avoid detecting this via the pmtmr calibration, and instead use a
>> DMI blacklist so we make sure we only trigger this on known problematic systems.
>> This also avoids the need for redefining existing boot time options (it would
>> just work :)
> 
> Nope, I doubt it would work (unless I misunderstood your comment) because the
> problem is intermittent (as described in the bug description). It means that
> sometimes the system boots right, sometimes it runs 3x too fast. So 
> blacklisting won't help here. Plus, most of those laptops just work fine,
> blacklisting them will lead to a bunch of problems.

I still believe the re-calibration is needed, however what I'm suggesting is
that that re-calibration only occurs on systems where we know this issue occurs.

This avoids the issue Andi pointed out where we wont' falsely trigger and
re-calibrate the PIT on a system w/ a bad ACPI PM timer, and additionally avoids
the need for boot time options.
Comment 70 Andi Kleen 2006-01-30 22:58:16 UTC
Created attachment 7186 [details]
Calibrate APIC timer using PM timer

I came up with this patch now. It calibrates the APIC timer using the PM
timer. it is relative to the firstfloor x86-64 tree and requires the
apic-main-timer patch in there.

So far the pmtimer calibration defaults to off, but can be enabled with
"apicpmtimer". Does it work with this patch and apicpmtimer?
Comment 71 Olivier Fourdan 2006-01-31 12:52:42 UTC
Created attachment 7190 [details]
Log of 2.6.16-rc1-mm4 with apic-main-timer.patch and apic-pm-timer.patch

Hi Andy,

I tried the patch and it doesn't fix the issue with the timer. The system clock
 still runs 3x times too fast even with "apicpmtimer" and moreover I now get a
bunch of soft lockups. Actulally the system hangs until I generate an event
using either the keyboard or the mouse.

Also, it seems that the CPU and bogomips calibration still use the PIT timer
which is wrong, so all these calibrations are also wrong (according to the
log).

PS: I really appreciate the time you and John spend on this.
Thanks
Olivier.
Comment 72 Olivier Fourdan 2006-02-13 03:29:29 UTC
Andy, I tried with the latest 2.6.16-rc2-mm1 with apicpmtimer and apicmaintimer
options set but the closk still runs too fast.

It seems your previous patches got merged (at least partially) so they don't
apply anymore and I'm not sure where I should start to investigate now.

Any hint would be greatly welcomed ;)
Comment 73 Andi Kleen 2006-02-13 03:40:52 UTC
Hmm, can you stick a printk into the pmtmr part in calibrate_APIC_clock
and double check it's really executed?

The frequency from your old log looks roughly ok:

Detected 12.481 MHz APIC timer.

My AMD boxes here have something in that ballpark too (e.g. 12.500Mhz 
on this machine, the small difference should not matter)

I assume you still see that frequency with the latest kernel, right?
With that i don't know what could be still wrong.

Anyways, i guess i need to implement a new timer mode anyways
for ATI boards with broken C2, perhaps that will help in your case too.
Comment 74 Olivier Fourdan 2006-03-20 13:02:28 UTC
Hi Andi,

With 2.6.16 and your latest fixes (ie using "apicpmtimer"), it works mostly
well. The clock runs at a correct speed except for some apps that still runs way
too fast (like games for examples). 

Do you think that be fixed too? My own patch that calibrates the PIT doesn't
address this issue either.

Cheers,
Olivier.
Comment 75 Brian Lindholm 2006-09-05 21:01:11 UTC
I realize that my comment is a little late, but I recently  discovered that
disabling "PCI Spread Spectrum" and "HT Spread Spectrum" fixed the double-speed
clock on my ASUS A8N-VM based system running an Athlon 64 X2 3800+ processor. 
Chipset is NVidia's nForce 410.  Running stock 2.6.17.11 kernel.  No extra patch
required.
Comment 76 Andi Kleen 2006-10-22 07:37:54 UTC
Ah i had nearly forgotten about this.

John, this means that apicmaintimer might be still needed and can't be dropped
in your rework.
Comment 77 Thomas Gleixner 2007-03-26 23:22:26 UTC
Andi,

can we close this one ?

    tglx
Comment 78 Olivier Fourdan 2007-03-27 01:18:13 UTC
> can we close this one ?

I am not Andy, but as far as I am concerned, yes, the hardware has finally died
for good and the issue went away once I replaced the motherboard in the laptop.
Comment 79 Alexandre Oliva 2007-03-27 03:21:06 UTC
This bug still affects my wife's notebook every other boot up.  Running FC6 with
latest updates, which means some 2.6.20 kernel.
Comment 80 Natalie Protasevich 2007-09-04 08:16:29 UTC
Any updates on this problem? Is it still there in 2.6.23+?
Thanks.
Comment 81 John Talbut 2007-10-31 12:07:11 UTC
Is anything happening about this?  Are there any workarounds?  I am still getting the 3x speed problem with kernel 2.6.28 on an AMD64 laptop.
Comment 82 John Talbut 2007-10-31 12:09:01 UTC
As in 2.6.18, I am not that up to date!
Comment 83 Ingo Molnar 2007-11-18 07:05:39 UTC
Could you try a more recent upstream kernel? 2.6.23 perhaps? 2.6.18 is not actively maintained anymore. (unless your distro does it)
Comment 84 Alexandre Oliva 2007-11-19 08:09:06 UTC
FWIW, we haven't observed this problem any more with 2.6.23.  I don't remember when we last hit it, unfortunately.
Comment 85 Natalie Protasevich 2008-03-03 22:17:00 UTC
John, did you get chance to try #87? Anyone else can confirm the bug with the latest kernel?
Comment 86 Natalie Protasevich 2008-05-02 16:58:12 UTC
I guess the bug is gone. Closing.
Comment 87 John Talbut 2008-08-06 00:15:01 UTC
I am afraid it has not.  I am now in kernel 2.6.25 and the problem is still happening.  Every time I boot the computer after it has been off for a while the clock is running at 3 times speed.  I have to do a cold restart and then it runs correctly.
Comment 88 John Talbut 2009-09-20 12:10:07 UTC
I am now on 2.6.30 and the problem is as bad as ever.
Comment 89 john stultz 2009-09-28 18:24:37 UTC
John Talbut: Can you provide any details on your specific hardware? Is it exactly the same as Olivier's? Can you provide dmesg / dmidecode info? Could you also provide output from:
 cat /sys/devices/system/clocksource/clocksource0/current_clocksource
Comment 90 John Talbut 2009-10-02 16:36:01 UTC
Oliver's:
Hardware Environment: HP Pavilion zv5000z notebook PC - AMD Athlon64 3000+, AMD
K8 Northbridge, nVidia nForce3 PCI bridge, 64 MB nVidia GeForce 4 440 Go
graphics, 768 MB PC2700 SODIMM...

Mine:
HP Compaq nx9105 notebook PC - AMD Athlon(tm) XP Processor 2800+,  AMD K8 Northbridge, nVidia Corporation nForce3 PCI Bridge, nVidia Corporation NV17 [GeForce4 420 Go 32M] graphics, 256MB SDRAM/DIMM

My software is Debian testing.

dmesg and dmidecode info attached.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
pit

I currently have clock=pit in my boot parameters, I have tried the other settings and they do not seem to make any difference.
Comment 91 John Talbut 2009-10-02 16:38:07 UTC
Created attachment 23230 [details]
nx9105 dmesg and dmidecode

Note You need to log in before you can comment on or make changes to this bug.