http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/atomicops-internals-x86.cc says " // Opteron Rev E has a bug in which on very rare occasions a locked // instruction doesn't act as a read-acquire barrier if followed by a // non-locked read-modify-write instruction. Rev F has this bug in // pre-release versions, but not in versions released to customers, // so we test only for Rev E, which is family 15, model 32..63 inclusive. if (strcmp(vendor, "AuthenticAMD") == 0 && // AMD family == 15 && 32 <= model && model <= 63) { AtomicOps_Internalx86CPUFeatures.has_amd_lock_mb_bug = true; } else { AtomicOps_Internalx86CPUFeatures.has_amd_lock_mb_bug = false; } " http://groups.google.com/group/google-perftools/browse_thread/thread/3d1b78d4a9db8c6e Solaris kernel for example has some workarounds for this bug.
From Mikael Pettersson mikpe / it.uu.se I investigated the Solaris track, but I've found no detailed explanation of the alleged bug. I've asked the Sun engineer who committed the fix for an explanation, but so far there's been no reply. Anyway, here's what I've found out. It's Solaris bug # 6323525. They call it "Mutex primitives don't work as expected." if (number_of_cores() < 2) then don't have bug if (family == 0xf && Model < 0x40) then have bug if (rdmsr(MSR_BU_CFG/*0xC0011023*/) & 2) then bug is masked lock: // mutex_lock, spin_lock, etc ... lock; cmpxchg .. jnz fail ret; nop; nop; nop // patched to "lfence; ret" if bug The workaround is to place a fencing instruction (lfence) between the mutex operation and the subsequent read-modify-write instruction. (This provides the necessary load memory barrier.) There's no change to the unlock code. Anyone know who to contact @ AMD about confirming or denying this?
From Elsie Wahlig elsie.wahlig / amd.com: Your issue may be one that has been seen on 1st generation AMD Opteron processor's with cpuid family 0Fh, cpuid model's < 40h with the code sequence that performs a read-modify write operation after acquiring a semaphore. The memory read ordering between a semaphore operation and a subsequent read-modify-write instruction (an instruction which uses the same memory location as both a source and destination) may allow the read-modify-write instruction to operate on the memory location ahead of the completion of the semaphore operation and an erratum may occur. If you think your software is encountering this code sequence, a work-around should be implemented by adding an LFENCE instruction right after the semaphore, after a cpuid check. The workaround's applied to OpenSolaris at http://mail.opensolaris.org/pipermail/onnv-notify/2006-October/009080.ht ml and Google performance tools tool at http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/at omicops-internals-x86.cc are suitable examples. A list of the model numbers this issue may occur on is at http://products.amd.com/en-us/downloads/AMD_Opteron_First_Generation_Ref erence_101607.pdf.
Most likely manifestation of this bug on a real system: http://bugs.mysql.com/bug.php?id=26081
Maybe Mark can help with this?
Thanks for the vote of confidence. Elsie's copied response is correct. http://mail.opensolaris.org/pipermail/onnv-notify/2006-October/009080.html and Google performance tools tool at http://google-perftools.googlecode.com/svn-history/r48/trunk/src/base/atomicops-internals-x86.cc are suitable examples. A list of the model numbers this issue may occur on is at http://products.amd.com/en-us/downloads/AMD_Opteron_First_Generation_Reference_101607.pdf.
One more note: there is no official errata for this bug. We tried to find it and failed.
Created attachment 20875 [details] x86: cleanup alternative.h Preparation cleanup patch.
Created attachment 20876 [details] x86 amd fix cmpxchg read acquire barrier That should add the missing lfence on the buggy amd cpus using the alternatives mechanism. It's kind of sad that we have to do such workarounds at the kernel level though. All userspace libs should also be audited (e.g. pthreads). Submitted as RFC, more testing would be welcome.
Do you think this is errata #147 or something different? Potential Violation of Read Ordering Rules Between Semaphore Operations and Unlocked Read-Modify-Write Instructions The description and workaround sounds similar. However the description states that the problem should not occur if the read-modify-write contains a lock prefix. The kernel code has had a lock prefix to cmpxchg for a long time.