Bug 219431

Created attachment 307076 [details]
/proc/cpuinfo

Created attachment 307077 [details]
/proc/config.gz

Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate dyndbg on the kernel command line for your system on 6.12rc5?

Created attachment 307079 [details]
dmesg w/ dynamic debug on amd-pstate.c/cppc.c/cppc_acpi.c

(In reply to Mario Limonciello (AMD) from comment #3)
> Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate
> dyndbg on the kernel command line for your system on 6.12rc5?

Reproduced on 6.12-rc5. There was nothing from amd_pstate.c before the warning message, so I also enabled dynamic debug on cppc.c and cppc_acpi.c.

It looks like this message is being emitted specifically from the calls in amd_set_max_freq_ratio().

You system is a shared memory design not a MSR design.
"[  +0.000016] amd_pstate: AMD CPPC shared memory based functionality is supported"

This looks like the calltrace:
amd_set_max_freq_ratio
->amd_get_boost_ratio_numerator
->->amd_detect_prefcore
->->->amd_get_highest_perf
->->->->cppc_get_highest_perf
->->->->->cppc_get_perf

The first error you see is "No CPC descriptor for CPU:1" which comes from cppc_get_perf().

Can you please share your acpidump?  It sure seems like a BIOS bug to me.

Created attachment 307080 [details]
acpidump, 5950X, ASUS Pro WS X570-ACE

I'm experiencing this issue too. Attaching acpidump output for 5950X and ASUS Pro WS X570-ACE motherboard with BIOS version 4805.

The latest BIOS for my M/B is 4902, but I haven't got a chance to update just yet.

Please also share the same dyndbg output.  Let's make sure we're talking about the same thing.

Are you guys seeing any functional issues you can elaborate upon?

Created attachment 307081 [details]
dmesg dyndbg, 5950X, ASUS Pro WS X570-ACE

Attached dmesg with dyndbg.

I do not think I can describe any functional limitations I face.

Created attachment 307082 [details]
`acpidump -b`, 5950X, ASRock X570M-Pro4, BIOS P3.60

Attaching `acpidump -b` output for my system, which is a 5950X on an ASRock X570M-Pro4 motherboard. The BIOS is also quite old here.

I'm not observing any specific functional limitations *that I can track to this issue*.

However, there is this thing that basically nothing I do to the EPP knobs or the cpufreq governor has any effect on the CPU frequency management behavior. Also, sometimes under ~75% load (cgroup limited) some cores get stuck at <4GHz while others boost to expected ~4.7-ish, but I've no idea how to reproduce this one reliably.

(In reply to Mario Limonciello (AMD) from comment #5)
> It looks like this message is being emitted specifically from the calls in
> amd_set_max_freq_ratio().
> 
> You system is a shared memory design not a MSR design.
> "[  +0.000016] amd_pstate: AMD CPPC shared memory based functionality is
> supported"

I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than MSR-based? That's why I mentioned Vermeer in both my original mail and this bug report.

> This looks like the calltrace:
> amd_set_max_freq_ratio
> ->amd_get_boost_ratio_numerator
> ->->amd_detect_prefcore
> ->->->amd_get_highest_perf
> ->->->->cppc_get_highest_perf
> ->->->->->cppc_get_perf
> 
> The first error you see is "No CPC descriptor for CPU:1" which comes from
> cppc_get_perf().
> 
> Can you please share your acpidump?  It sure seems like a BIOS bug to me.

There is also an interesting observation, that at some point *after* this warning we get a pr_debug() from the end of amd_detect_prefcore(), which indicates that the latter is called again and this error does not happen anymore:

```
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1
Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19)
<...>
Oct 28 17:13:08 kernel: amd_pstate: AMD CPPC shared memory based functionality is supported
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: AMD CPPC preferred core is supported (highest perf: 0xe7)
```

It's even more interesting that the debug logs from the amd-pstate flow you described earlier appear to be interleaved with debug logs from acpi_cppc_processor_probe():

```
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1
Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19)
Oct 28 17:13:08 kernel: Monitor-Mwait will be used to enter C-1 state
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 1 <-- THIS
```

I'm not familiar with control flow in these subsystems, but isn't it possible that we are simply racing with ACPI (and thus CPPC) subsystem initialization?

>I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than
>MSR-based? That's why I mentioned Vermeer in both my original mail and this
>bug report.

Vermeer is a shared memory based design.  All the newer designs are MSR based.

> There is also an interesting observation, that at some point *after* this
> warning we get a pr_debug() from the end of amd_detect_prefcore(), which
> indicates that the latter is called again and this error does not happen
> anymore:

Yeah that's why I was wondering if there was a functional issue. It seems like this is going to influence some of the frequency invariance code that's set up earlier on.

> I'm not familiar with control flow in these subsystems, but isn't it possible
> that we are simply racing with ACPI (and thus CPPC) subsystem initialization?

acpi_init() calls 
acpi_bus_init()() calls 
acpi_bus_osc_negotiate_platform_control() 
which sets osc_sb_cppc2_support_acked

Here is the more full call path that fails:

acpi_init() calls
acpi_scan_init() calls
acpi_processor_init() 

which loads "processor" kernel module

From "processor" kernel module:

acpi_processor_driver_init() calls
acpi_soft_cpu_online() calls
__acpi_processor_start() calls 
acpi_cppc_processor_probe() calls 
arch_init_invariance_cppc() which is a macro to call 
init_freq_invariance_cppc().


I'm not sure I understand why CPPC v2 _OSC isn't acked though at the right time.  

Can you use ftrace or annotate the functions with some printk's to figure it out perhaps from what I've shared?

Something else that would be really useful to understand is whether this is only showing up now because the error level was raised
(IE : https://git.kernel.org/superm1/c/21fb59ab4b976) or if it was there in past kernels too, but because it was debug no one ever saw it.

Try turning up dynamic debug on arch/x86/kernel/acpi/cppc.c on an older kernel that doesn't have that commit and see if you see it there too.

I suspect the answer is yes.

> I suspect the answer is yes.

Actually; no it should be new, but it's because the frequency calculation was wrong before.

I found a system with shared mem and I have a theory what's going on is that frequency invariance is attempting to get setup before all CPUs have probed.

Before the preferred cores code frequency invariance would be calculated from the first CPU only as _CPC is probed.  But with preferred cores code in place you need ALL _CPC values present to know whether you have preferred cores or not.

I added some dump stack calls to acpi_cppc_processor_probe when the struct is parsed:

ACPI CPPC: Parsed CPC struct for CPU: 0
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd #268
Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024
Call Trace:
 <TASK>
 dump_stack_lvl+0x60/0x80
 acpi_cppc_processor_probe+0x9fc/0xe50
 acpi_soft_cpu_online+0x69/0x160
 ? __pfx_acpi_soft_cpu_online+0x10/0x10
 cpuhp_invoke_callback+0x308/0x4c0
 ? srso_return_thunk+0x5/0x5f
 ? __pfx_smpboot_thread_fn+0x10/0x10
 cpuhp_thread_fun+0x131/0x210
 smpboot_thread_fn+0x187/0x220
 kthread+0xdd/0x110
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x30/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>

Then I added one to the error sequence.

ACPI CPPC: No CPC descriptor for CPU:1
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd #268
Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024
Call Trace:
 <TASK>
 dump_stack_lvl+0x60/0x80
 cppc_get_perf+0x127/0x1c0
 amd_get_highest_perf+0x5e/0x80
 amd_detect_prefcore+0x7a/0x130
 amd_get_boost_ratio_numerator+0x28/0x90
 init_freq_invariance_cppc+0xad/0x140
 acpi_cppc_processor_probe+0x7ce/0xe50
 acpi_soft_cpu_online+0x69/0x160
 ? __pfx_acpi_soft_cpu_online+0x10/0x10
 cpuhp_invoke_callback+0x308/0x4c0
 ? srso_return_thunk+0x5/0x5f
 ? __pfx_smpboot_thread_fn+0x10/0x10
 cpuhp_thread_fun+0x131/0x210
 smpboot_thread_fn+0x187/0x220
 kthread+0xdd/0x110
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x30/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Could not retrieve highest performance (-19)

You can see that in both cases the code runs on CPU0.  So it's trying to get values that won't be initialized until CPU1 is ready.

I think this will help "prevent" the error from coming up:

diff --git a/arch/x86/kernel/acpi/cppc.c b/arch/x86/kernel/acpi/cppc.c
index 5e9325273a9fb..a21c26d1fe65a 100644
--- a/arch/x86/kernel/acpi/cppc.c
+++ b/arch/x86/kernel/acpi/cppc.c
@@ -190,7 +190,7 @@ int amd_detect_prefcore(bool *detected)
                break;
        }

-       for_each_present_cpu(cpu) {
+       for_each_online_cpu(cpu) {
                u32 tmp;
                int ret;

But it's going to probably break preferred cores detection, so it's not a correct solution.

Created attachment 307084 [details]
possible patch (v1)

Here's a potential patch.  This helps on the system I can reproduce this issue, please see if helps you too.

Also if it does help, would you mind if I used your email address in the commit message for the Reported-and-tested-by tag?  I can get it from Bugzilla, but I would like your permission first.

I'll test this later today, thanks. (I also want to try to dig into _OSC CPCv2 flag not being set, just to definitively confirm if this is a firmware issue or an ordering issue.)

The message disappears with this patch for me. Should I check something else too?

Thanks, you can check whether the CPU cores are reporting correct maximum frequencies.

Can I use your email when I post it?

> you can check whether the CPU cores are reporting correct maximum
> frequencies.

In `amd_pstate_max_freq` it's 5084000 on all cores. Running `turbostat` parallel to `xmrig` I see ~4.4 GHz on all cores under heavy load. In this regard, for my setup, nothing changed comparing to an unpatched kernel, I guess.

> Can I use your email when I post it?

Sure.

https://lore.kernel.org/linux-acpi/20241029174910.600482-1-superm1@kernel.org/

(In reply to Mario Limonciello (AMD) from comment #14)
> Created attachment 307084 [details]
> possible patch (v1)
> 
> Here's a potential patch.  This helps on the system I can reproduce this
> issue, please see if helps you too.

I can confirm too that this patch fixes the error and *I think* actually makes amd-pstate respond to governor/EPP changes for me, but I did not test that rigorously.

(In reply to Mario Limonciello (AMD) from comment #11)
> I'm not sure I understand why CPPC v2 _OSC isn't acked though at the right
> time.  
> 
> Can you use ftrace or annotate the functions with some printk's to figure it
> out perhaps from what I've shared?

I investigated this as well, and it turns out that the relevant _OSC simply fails to evaluate on this motherboard:

[  +0,000000] ACPI: Added _OSI(Module Device)                                                                                                                                                                                                               [  +0,000002] ACPI: Added _OSI(Processor Device)                                                                                                                                                                                                            [  +0,000002] ACPI: Added _OSI(3.0 _SCP Extensions)                                                                                                                                                                                                         [  +0,000001] ACPI: Added _OSI(Processor Aggregator Device)                                                                                                                                                                                                 [  +0,007541] ACPI: 8 ACPI AML tables successfully acquired and loaded                                                                                                                                                                                      [  +0,002668] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored                                                                                                                                                                                          [  +0,001329] ACPI: \_SB_: _OSC: OS supports [ProcessorAggregator PPC-OST _PR3 HotplugOST APEI CPPC CPPCv2 PC-LPI FastThermalSampling 16+PStates GenericEventDevice CPPCDiverseHighest InterruptResourceSource CPPCFlexibleAddressSpace GenericInitiator USB4 BatteryChargeLimiting PRM FFHOpRegion]
[  +0,000000] ACPI: \_SB_: _OSC: failed to query platform: AE_NOT_FOUND

(I added the two last messages)

Where should I report this? Or is this 100% an issue with my motherboard? Or...?

> Where should I report this? Or is this 100% an issue with my motherboard?
> Or...?

It sounds like your BIOS just doesn't support it.

> I can confirm too that this patch fixes the error and *I think* actually
> makes amd-pstate respond to governor/EPP changes for me, but I did not test
> that rigorously.

OK. I submitted it upstream but the build bot hit some problems with v2.

https://lore.kernel.org/linux-pm/67290cbd.630a0220.3b06d1.05d1@mx.google.com/

I'll need to figure them out and resubmit a v3.