LWP = Lightweight Profiling: http://developer.amd.com/wordpress/media/2012/10/43724.pdf On an AMD FX-8320E CPU, the first CPU core in a module has LWP disabled, while the second one has it enabled. This creates problems when using GCC with "-march=native" (which enables or not "-mlwp", depending on which core it runs) and mixing the resulting LWP and non-LWP objects: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86007 Diagnostic commands: $ grep -E '(core id|lwp)' /proc/cpuinfo core id : 0 core id : 1 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold core id : 2 core id : 3 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold core id : 4 core id : 5 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold core id : 6 core id : 7 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold $ cpuid | grep 'lightweight profiling support' lightweight profiling support = false lightweight profiling support = true lightweight profiling support = false lightweight profiling support = true lightweight profiling support = false lightweight profiling support = true lightweight profiling support = false lightweight profiling support = true
Upload .config and full dmesg pls. Also output of "cpuid -r" pls. I'm assuming this is baremetal box, not a guest, etc. Also, do you have the latest BIOS from your mobo vendor installed? Thx.
Created attachment 301388 [details] 5.17.9 dmesg -T
Created attachment 301389 [details] 5.17.9 .config
Created attachment 301390 [details] cpuid -r
Yes, this is bare metal. The motherboard has the latest BIOS and the CPU has the latest microcode.
0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01eb3fff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01ebbfff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01eb3fff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01ebbfff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01eb3fff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01ebbfff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01eb3fff edx=0x2fd3fbff 0x80000001 0x00: eax=0x00600f20 ebx=0x10000000 ecx=0x01ebbfff edx=0x2fd3fbff well, the kernel is basically reporting what the hardware (CPUID) says... Looking at your BIOS: [Fri Jul 8 01:55:57 2022] DMI: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016 I don't think you'll get a new one. So unless Mario has a better idea, I don't see what we can do here. Unless maybe you boot with clearcpuid=lwp and you don't use LWP at all. It'll fix your compilation at least. HTH.
I guess there is the possibility that we can flag this CPU and setup a quirk to disable lwp automatically if not all the cores agree (like what that kernel command line option would do).
I'd say "this BIOS" because I don't think the CPU is wrong here - BIOS is likely not setting up things properly or so.
It's the microcode. If I disable early microcode loading by disabling CONFIG_MICROCODE_AMD, thus no longer updating 0x6000822 to 0x06000852, LWP is present on all cores. There's an intriguing piece of info in this 2019 Xen commit message - https://patchwork.kernel.org/project/xen-devel/patch/1558347216-19179-1-git-send-email-andrew.cooper3@citrix.com/ : "LWP was dropped from Fam15/16 CPUs when IBPB for Spectre v2 was introduced in microcode, owing to LWP not being used in practice." Could we be looking at a botched feature disabling attempt?
> Unless maybe you boot with clearcpuid=lwp and you don't use LWP at all. It'll > fix your compilation at least. That would work, since I only need predictable "-march=native" output, not LWP, but adding "clearcpuid=207" to the 5.17.9 command line only disabled the feature flag in /proc/cpuinfo. Both cpuid and gcc still see it on half the cores. Anyway, my long-term workaround is to pass "-no-lwp" to gcc, but not knowing what was going on under the hood bothered me.
(In reply to Ștefan Talpalaru from comment #10) > That would work, since I only need predictable "-march=native" output, not > LWP, but adding "clearcpuid=207" to the 5.17.9 command line only disabled > the feature flag in /proc/cpuinfo. Yeah, the feature-name thing is in 5.19. > Both cpuid and gcc still see it on half the cores. Yeah, this is correct - it won't help you if your compiler does the detection itself as the kernel cannot clear the CPUID flag. Maybe... > Anyway, my long-term workaround is to pass "-no-lwp" to gcc, but not knowing > what was going on under the hood bothered me. Yeah, lemme dig more about the ucode thing. Thx.
Can you read the ext feature mask on all cores? modprobe msr rdmsr -a 0xc0011005
Yap, we might be able to fix the intention of the BIOS fix through those. :-)
> rdmsr -a 0xc0011005 1eb3fff2fd3fbff 1ebbfff2fd3fbff 1eb3fff2fd3fbff 1ebbfff2fd3fbff 1eb3fff2fd3fbff 1ebbfff2fd3fbff 1eb3fff2fd3fbff 1ebbfff2fd3fbff
Thanks, and now, does `wrmsr -a 0xc0011005 0x1eb3fff2fd3fbff` resolve your issue?
> does `wrmsr -a 0xc0011005 0x1eb3fff2fd3fbff` resolve your issue? Yes, it disables LWP on all cores.
Right, I guess we can do that. It probably should go into init_amd_bd() and should be in a separate function, similar to clear_rdrand_cpuid_bit(). Just in case someone else reading here wants to do it... :-)
I found a similarly impacted system down the back of the sofa and had a play. (HP ProLiant DL385 G7 with firmware from 03/19/2012 if we're keeping score :) ) When we load microcode on CPU0, we lose the LWP feature bit (both CPUID and in the mask MSR) and gain the IBPB bit. This is as expected. I had a bit of a play, and I can't figure out what else actually got disabled. All other CPUID details pertaining to LWP remain identical, you can still set %xcr0.lwp and seemingly use the LWP instructions and MSRs, although I stopped short of actually turning it on properly and checking what got written into the buffer. Then when CPU1 comes up, we read the microcode version, see that it's up to date, and skip the ucode load. Interestingly, while the LWP bit is still set, the IBPB bit is set too. So clearly the CPUID data from 0x80000008 (where IBPB lives) is coming from a shared location in the compute unit, whereas CPUID data for 0x80000001 (where LWP lives) is coming from a cluster-local location. This isn't surprising. The ability to control some of the CPUID data is a supported feature for hypervisors, and also the reason why I never encountered this bug with Xen. We context switch the CPUID MSRs per VM, and the heterogeneous levelling logic on boot has caused LWP to fall out of the default profiles. Anyway, if I force load microcode on CPU1 (i.e. ignore the version check), then the LWP CPUID bit does hide itself. I think we probably want to fix this by disregarding the version check for non-primary cluster members, because who knows what other per-CPU state the microcode (didn't) fix up.
Does the attached, only build-tested patch fix your issue? Thx.
Created attachment 301560 [details] test fix Remove ucode revision checking.
Yes, the patch disables LWP on all cores. Tested with a Git HEAD kernel.
Thanks for testing, lemme do a proper patch.
ILL this problem exists only with AMD Bulldozer Family 15h microprocessor micro-architecture https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture) with "Clustered Multithreading" (CMT), which is distinct from hyperthreading https://en.wikipedia.org/wiki/Hyper-threading AMD calls this design a "Module". 1 module = 2 integer + 1 FPU cores.
> only with AMD Bulldozer Family Bulldozer, Piledriver, Steamroller and Excavator are all affected.
(In reply to Ștefan Talpalaru from comment #24) > > only with AMD Bulldozer Family > > Bulldozer, Piledriver, Steamroller and Excavator are all affected. Yes, whole AMD "Heavy equipment" family (Bulldozer-based). Maybe there'll be some benefit for multi-kernel hyper-threaded AMD Zen-based CPUs in using previous code path?
Fix on its way upstream: df76acb227a0 ("x86/microcode/AMD: Attempt applying on every logical thread") We're done here.
Turns out this is not as easy - I've retracted this fix as it is a bit lacking in some situations. A better solution is forthcoming. Thx.
Ok, I have had some time finally to do a second version of the fix. Pls give it a try. Thx.
Created attachment 302939 [details] Test fix v2 Second version of the fix, just early patching.
Second patch tested and working as advertised. LWP is now disabled on all cores.
Thanks, I'll add your Tested-by and queue it along with a couple more fixes.