Bug 10585 - System Timer can come up bad
Summary: System Timer can come up bad
Status: CLOSED CODE_FIX
Alias: None
Product: Platform Specific/Hardware
Classification: Unclassified
Component: x86-64 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: platform_x86_64@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-05-01 09:28 UTC by Charles Mitchell
Modified: 2008-09-04 13:37 UTC (History)
4 users (show)

See Also:
Kernel Version: 2.6.25
Subsystem:
Regression: ---
Bisected commit-id:


Attachments

Description Charles Mitchell 2008-05-01 09:28:56 UTC
Hardware: Intel S5000PAL/PSL motherboard, dual 5160 and dual E5450. 
          PCIe plugin card to IDT Switch
 
Software Environment:  

Problem Description:  We noticed that some of our systems were coming up with bad system time, i.e. cpu_khz in /proc/cpuinfo was wrong and time was not being correctly kept(losing several seconds per minute).  This problem happens more often on some machines than others.  The problem is due to the fact that the number of cpu cycles over 30 ms is occassionally too large and the cpu_khz calculated in tsc.c ends up too large.  I suspect that an NMI is sneaking in, delaying the time at which the OUT pin is sampled high (inb_p(0x61) & 0x20). 

The 2.6.25 code is the same as 2.6.18 for the sampling of the OUT pin.  

Test results:

   kernel           result
============      =================
2.6.18.solace     Fails approx 1 in 3 reboots    
2.6.18.8          Failed after 16 reboots
2.6.25            2 fails in 244 reboots

The interesting thing is that 2.6.25 had an extremely rare "cpu looks slow"
event.  Normally the cpu looks too fast because cpu_cycles are recorded while
the cpu is distracted with an NMI.  In the too slow case - the only way this
can happen is if the NMI happens at a very specific time - in between
programming the timer and reading the 'start' cycle count. So in this case we
know extactly how long the NMI was - it was 5.84 ms.

We plan on fixing by repeating the measurement some number of times and selecting the best results.

bogomips are also independently calculated incorrectly - this is a separate problem that I believe is of little consequence.

Steps to reproduce:   reboot and look at /proc.cpuinfo for incorrect CPU MHz.
Comment 1 Andrew Morton 2008-05-01 09:37:56 UTC
Reassigning to platform/x86_64.

ISTR this was reported and discussed a couple of months
ago, but I don't recall any conclusion.

Could it be an SMI?
Comment 2 Jike Song 2008-05-06 22:40:09 UTC
is there any update on this BUG? I'm suffering it also.
Comment 3 Jike Song 2008-05-07 00:05:49 UTC
Sorry, my bad.  It's only an incorrect KDE setting for me. 
Comment 4 Thomas Gleixner 2008-05-07 01:48:47 UTC
(In reply to comment #1)
> Reassigning to platform/x86_64.
> 
> ISTR this was reported and discussed a couple of months
> ago, but I don't recall any conclusion.
> 
> Could it be an SMI?

Yes, we have seen this in apic timer calibration as well. What I wonder is why this happens on 64bit. The 64 bit code has a detection for that already when TSC/cpu_khz is calibrated.
Comment 5 Charles Mitchell 2008-05-07 12:07:03 UTC
The code in arch/i386/kernal/tsc.c has no checks for failures - there is a one-time reading of a start/end pair.

In init/calibrate.c there is an attempt to catch bad readings and not incude them in an average - but that is for bogomips.
Comment 6 Thomas Gleixner 2008-05-08 05:59:40 UTC
> ------- Comment #5 from charles.mitchell@solacesystems.com  2008-05-07 12:07
> -------
> The code in arch/i386/kernal/tsc.c has no checks for failures - there is a
> one-time reading of a start/end pair.
> 
> In init/calibrate.c there is an attempt to catch bad readings and not incude
> them in an average - but that is for bogomips.

Yeah, I know. I planned to unify tsc_32/64, but I did not come around
yet. Maybe that bug report is a good enough priority boost to go for
it.

Thanks,
	tglx
Comment 7 Thomas Gleixner 2008-09-04 13:37:10 UTC
Fixed in mainline: fbb16e243887332dd5754e48ffe5b963378f3cd2

Note You need to log in before you can comment on or make changes to this bug.