Bug 11305 - Opteron Rev E - missing workaround for a processor bug
Summary: Opteron Rev E - missing workaround for a processor bug
Status: CLOSED OBSOLETE
Alias: None
Product: Platform Specific/Hardware
Classification: Unclassified
Component: x86-64 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: platform_x86_64@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-08-11 09:46 UTC by Arkadiusz Miskiewicz
Modified: 2012-05-22 12:29 UTC (History)
4 users (show)

See Also:
Kernel Version: 2.6.27-git
Subsystem:
Regression: No
Bisected commit-id:


Attachments
x86: cleanup alternative.h (4.67 KB, patch)
2009-04-08 06:09 UTC, Mathieu Desnoyers
Details | Diff
x86 amd fix cmpxchg read acquire barrier (9.35 KB, patch)
2009-04-08 06:12 UTC, Mathieu Desnoyers
Details | Diff

Description Arkadiusz Miskiewicz 2008-08-11 09:46:39 UTC
http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/atomicops-internals-x86.cc
says

"  // Opteron Rev E has a bug in which on very rare occasions a locked
  // instruction doesn't act as a read-acquire barrier if followed by a
  // non-locked read-modify-write instruction.  Rev F has this bug in 
  // pre-release versions, but not in versions released to customers,
  // so we test only for Rev E, which is family 15, model 32..63 inclusive.
  if (strcmp(vendor, "AuthenticAMD") == 0 &&       // AMD
      family == 15 &&
      32 <= model && model <= 63) {
    AtomicOps_Internalx86CPUFeatures.has_amd_lock_mb_bug = true;
  } else {
    AtomicOps_Internalx86CPUFeatures.has_amd_lock_mb_bug = false;
  }
"

http://groups.google.com/group/google-perftools/browse_thread/thread/3d1b78d4a9db8c6e

Solaris kernel for example has some workarounds for this bug.
Comment 1 Arkadiusz Miskiewicz 2008-08-11 09:47:55 UTC
From Mikael Pettersson mikpe / it.uu.se

I investigated the Solaris track, but I've found no detailed
explanation of the alleged bug. I've asked the Sun engineer
who committed the fix for an explanation, but so far there's
been no reply.

Anyway, here's what I've found out.

It's Solaris bug # 6323525.

They call it "Mutex primitives don't work as expected."

if (number_of_cores() < 2) then don't have bug
if (family == 0xf && Model < 0x40) then have bug
if (rdmsr(MSR_BU_CFG/*0xC0011023*/) & 2) then bug is masked

lock:   // mutex_lock, spin_lock, etc
        ...
        lock; cmpxchg ..
        jnz fail
        ret; nop; nop; nop      // patched to "lfence; ret" if bug

The workaround is to place a fencing instruction (lfence) between
the mutex operation and the subsequent read-modify-write instruction.
(This provides the necessary load memory barrier.)

There's no change to the unlock code.

Anyone know who to contact @ AMD about confirming or denying this?
Comment 2 Arkadiusz Miskiewicz 2008-08-11 09:48:44 UTC
From Elsie Wahlig elsie.wahlig / amd.com:

Your issue may be one that has been seen on 1st generation 
AMD Opteron processor's with cpuid family 0Fh, cpuid model's 
< 40h with the code sequence that performs a read-modify write 
operation after acquiring a semaphore.  

The memory read ordering between a semaphore operation and a 
subsequent read-modify-write instruction (an instruction which 
uses the same memory location as both a source and destination) 
may allow the read-modify-write instruction to operate on the 
memory location ahead of the completion of the semaphore 
operation and an erratum may occur.    

If you think your software is encountering this code sequence, 
a work-around should be implemented by adding an LFENCE 
instruction right after the semaphore, after a cpuid check.    
The workaround's applied to OpenSolaris at 
http://mail.opensolaris.org/pipermail/onnv-notify/2006-October/009080.ht
ml 
and Google performance tools tool at 
http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/at
omicops-internals-x86.cc 
are suitable examples.  
A list of the model numbers this issue may occur on is at 
http://products.amd.com/en-us/downloads/AMD_Opteron_First_Generation_Ref
erence_101607.pdf.
Comment 3 Arkadiusz Miskiewicz 2008-08-11 09:51:01 UTC
Most likely manifestation of this bug on a real system:
http://bugs.mysql.com/bug.php?id=26081
Comment 4 Andrew Morton 2008-08-11 17:39:19 UTC
Maybe Mark can help with this?
Comment 5 Mark Langsdorf 2008-08-12 07:40:43 UTC
Thanks for the vote of confidence.  Elsie's copied response is correct.

http://mail.opensolaris.org/pipermail/onnv-notify/2006-October/009080.html 
and Google performance tools tool at 
http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/atomicops-internals-x86.cc 
are suitable examples.  
A list of the model numbers this issue may occur on is at 
http://products.amd.com/en-us/downloads/AMD_Opteron_First_Generation_Reference_101607.pdf.
Comment 6 Arkadiusz Miskiewicz 2008-08-12 08:07:14 UTC
One more note: there is no official errata for this bug. We tried to find it and failed.
Comment 7 Mathieu Desnoyers 2009-04-08 06:09:41 UTC
Created attachment 20875 [details]
x86: cleanup alternative.h

Preparation cleanup patch.
Comment 8 Mathieu Desnoyers 2009-04-08 06:12:38 UTC
Created attachment 20876 [details]
x86 amd fix cmpxchg read acquire barrier

That should add the missing lfence on the buggy amd cpus using the alternatives mechanism. It's kind of sad that we have to do such workarounds at the kernel level though. All userspace libs should also be audited (e.g. pthreads).

Submitted as RFC, more testing would be welcome.
Comment 9 Eric Jones 2009-05-21 19:18:50 UTC
Do you think this is errata #147 or something different? 

Potential Violation of Read Ordering Rules Between Semaphore Operations and Unlocked Read-Modify-Write Instructions

The description and workaround sounds similar. However the description states that the problem should not occur if the read-modify-write contains a lock prefix. The kernel code has had a lock prefix to cmpxchg for a long time.

Note You need to log in before you can comment on or make changes to this bug.