Bug 215800

So, now I have this and all the cores are stuck at 2.2GHz:

/sys/devices/system/cpu/cpu0/cpufreq:

amd_pstate_highest_perf:166
scaling_min_freq:550000
scaling_available_governors:conservative ondemand userspace powersave performance schedutil 
scaling_governor:ondemand
cpuinfo_max_freq:4929000
amd_pstate_lowest_nonlinear_freq:1752000
amd_pstate_max_freq:4929000
related_cpus:0
scaling_cur_freq:2190255
scaling_setspeed:<unsupported>
affected_cpus:0
scaling_max_freq:4929000
cpuinfo_transition_latency:131072
scaling_driver:amd-pstate
cpuinfo_min_freq:550000

Huang Rui,

Could you check, please?

Hi Artem,

Thanks for the report. Can you print: "lscpu | grep cppc"?

> Thanks for the report. Can you print: "lscpu | grep cppc"?

# lscpu | grep -i cppc
returns nothing.

# lscpu --version
lscpu from util-linux 2.37.4

I've loading the amd-pstate driver prior to running it.

"I've tried to fix the issue by `cat cpuinfo_max_freq > scaling_max_freq` but that didn't work. "

cat does not work. Please use "echo $number > scaling_max_freq".

(In reply to Joe from comment #5)
> "I've tried to fix the issue by `cat cpuinfo_max_freq > scaling_max_freq`
> but that didn't work. "
> 
> cat does not work. Please use "echo $number > scaling_max_freq".

From the original post:

> For my Ryzen 7 5800X I've tried (under root) numbers from 600 000 to 2 200
> 000 and they all work:
> 
> echo $number | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 
> However anything above 2 200 000 instead makes the CPU stay at 2.2GHz.

Echoing the number manually didn't work either.

Created attachment 300747 [details]
Max freq

Attach is the operation step on my side. It works with root privilege.

(In reply to Joe from comment #7)
> Created attachment 300747 [details]
> Max freq

Changing scaling_max_freq makes the CPU stay at frequencies 2.2GHz or below.

Enter e.g. 3000000 (3GHz) as scaling_max_freq for all cores, then observe that scaling_cur_freq never exceeds 2.2GHz.

Even restoring scaling_max_freq by setting it to cpuinfo_max_freq doesn't fix the issue.

Again, I'm repeating what I already said in the original bug report.



> For my Ryzen 7 5800X I've tried (under root) numbers from 600 000 to 2 200
> 000 and they all work:
> 
> echo $number | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 
> However anything above 2 200 000 instead makes **the CPU stay at 2.2GHz**.

Don't mind scaling_max_freq, it's scaling_cur_freq that never exceeds 2.2GHz!

OK, so it seems you change the scaling_max_freq to 2.2G, after that, even you change back to 3G or cpuinfo_max_freq, the scaling_cur_freq is still less than 2.2GHz.

I tried 2 devices on my local, can not reproduce this issue.

After you set scaling_max_freq to 3G or higher, You can try to run "stress --cpu 128" with increasing the workload to check if the scaling_cur_freq can exceeds 2.2GHz.

(In reply to Joe from comment #11)
> OK, so it seems you change the scaling_max_freq to 2.2G, after that, even
> you change back to 3G or cpuinfo_max_freq, the scaling_cur_freq is still
> less than 2.2GHz.
> 
> I tried 2 devices on my local, can not reproduce this issue.
> 
> After you set scaling_max_freq to 3G or higher, You can try to run "stress
> --cpu 128" with increasing the workload to check if the scaling_cur_freq can
> exceeds 2.2GHz.

I've run tests now a dozen times already: once I change cpuinfo_max_freq my CPU never goes above 2.2GHz no matter what I do even if I restore the value stored in this variable/file.

This is further corroborated by more than halved performance and decreased power consumption - it doesn't go above ~55W.

Created attachment 300750 [details]
/sys/devices/system/cpu/cpufreq

The entire /sys/devices/system/cpu/cpufreq tar.bz2'ed after running an intensive MT task for a few minutes.

Here's another weirdness/quirk:

After suspending and resuming the maximum CPU frequency increases to 3800MHz (which is the base frequency for Ryzen 7 5800X).

Boost is _enabled_.

Not sure if it warrants yet another bug report.

Let's limit them to three.

(In reply to Artem S. Tashkinov from comment #0)
> For my Ryzen 7 5800X I've tried (under root) numbers from 600 000 to 2 200
> 000 and they all work:
> 
> echo $number | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 
> However anything above 2 200 000 instead makes the CPU stay at 2.2GHz.
> 
> This is after booting with acpi-cpufreq and loading amd-pstate with
> replace=1.
> 
> Even switching to the performance governor keeps all the cores at 2.2GHz.
> 
> I've also tried the userspace governor and again only frequencies up to
> 2.2GHz work. Anything above will be ignored.
> 
> I've tried to fix the issue by `cat cpuinfo_max_freq > scaling_max_freq` but
> that didn't work.
> 
> That looks like a bug.

Could you help to try the below kernel ?
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/drivers/cpufreq/amd-pstate.c?h=next-20220516

I tried this kernel, but 5950x cannot reproduce the issue as yours.

(In reply to perry_yuan from comment #15)
> 
> Could you help to try the below kernel ?
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/
> drivers/cpufreq/amd-pstate.c?h=next-20220516
> 
> I tried this kernel, but 5950x cannot reproduce the issue as yours.

Does this patch require enabling debugging in the kernel? I'm not sure how I am supposed to share (debug) data once I've applied it.

(In reply to Artem S. Tashkinov from comment #16)
> (In reply to perry_yuan from comment #15)
> > 
> > Could you help to try the below kernel ?
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/
> > drivers/cpufreq/amd-pstate.c?h=next-20220516
> > 
> > I tried this kernel, but 5950x cannot reproduce the issue as yours.
> 
> Does this patch require enabling debugging in the kernel? I'm not sure how I
> am supposed to share (debug) data once I've applied it.

Hi Artem.
I mean you could build the kernel and install it on your system.
and switch the governor , capture the min/max log ,etc.

I do not find the reproducing way, but I still want to try to reproduce the issue if you can confirm the kernel link has same problem.

Perry.

(In reply to Artem S. Tashkinov from comment #0)
> For my Ryzen 7 5800X I've tried (under root) numbers from 600 000 to 2 200
> 000 and they all work:
> 
> echo $number | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 
> However anything above 2 200 000 instead makes the CPU stay at 2.2GHz.
> 
> This is after booting with acpi-cpufreq and loading amd-pstate with
> replace=1.
> 
> Even switching to the performance governor keeps all the cores at 2.2GHz.
> 
> I've also tried the userspace governor and again only frequencies up to
> 2.2GHz work. Anything above will be ignored.
> 
> I've tried to fix the issue by `cat cpuinfo_max_freq > scaling_max_freq` but
> that didn't work.
> 
> That looks like a bug.

I would like to confirm which kernel version or patch cause the regression,
so need you help to verify the regression root cause, then I will reproduce the issue to fix it on my side.

 
Perry

All the same with 5.18.3.

Steps to reproduce:

echo 3000000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

Now all the CPU cores are stuck below 2.2GHz.

This is after `modprobe amd-pstate replace=1`

I really don't understand what else I can do. This is reproducible with a single command on my Ryzen 7 5800X CPU.

> capture the min/max log

Now idea how to do that, where is it logged.

/sys/devices/system/cpu/cpu0/acpi_cppc contains this:

feedback_ctrs:ref:1185870842796 del:939045433401
lowest_nonlinear_perf:59
highest_perf:186
nominal_freq:3801
lowest_freq:550
nominal_perf:128
lowest_perf:19
reference_perf:128
wraparound_time:18446744073709551615

No idea what highest_perf is - it's a read only value anyways. No idea what its scale is, 0-255? Why is so below the maximum value?

/sys/devices/system/cpu/cpufreq/policy0

amd_pstate_highest_perf:166
scaling_min_freq:550000
scaling_available_governors:conservative ondemand userspace powersave performance schedutil 
scaling_governor:ondemand
cpuinfo_max_freq:4929000
amd_pstate_lowest_nonlinear_freq:1752000
amd_pstate_max_freq:4929000
related_cpus:0
scaling_cur_freq:1247422
scaling_setspeed:<unsupported>
affected_cpus:0
scaling_max_freq:1600000
cpuinfo_transition_latency:131072
scaling_driver:amd-pstate
cpuinfo_min_freq:550000

Why do I have two wildly different amd_pstate_highest_perf=166 and highest_perf=186?

What's the relationship between the two?

Anyways, the issue remains. As far as I can see https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/drivers/cpufreq/amd-pstate.c has seen near zero activity for the past three months.

(In reply to Artem S. Tashkinov from comment #21)
> /sys/devices/system/cpu/cpufreq/policy0
> 
> amd_pstate_highest_perf:166
> scaling_min_freq:550000
> scaling_available_governors:conservative ondemand userspace powersave
> performance schedutil 
> scaling_governor:ondemand
> cpuinfo_max_freq:4929000
> amd_pstate_lowest_nonlinear_freq:1752000
> amd_pstate_max_freq:4929000
> related_cpus:0
> scaling_cur_freq:1247422
> scaling_setspeed:<unsupported>
> affected_cpus:0
> scaling_max_freq:1600000
> cpuinfo_transition_latency:131072
> scaling_driver:amd-pstate
> cpuinfo_min_freq:550000
> 
> Why do I have two wildly different amd_pstate_highest_perf=166 and
> highest_perf=186?
> 
> What's the relationship between the two?
> 
> Anyways, the issue remains. As far as I can see
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/
> drivers/cpufreq/amd-pstate.c has seen near zero activity for the past three
> months.
 modprobe amd-pstate replace=1(In reply to Artem S. Tashkinov from comment #19)
> All the same with 5.18.3.
> 
> Steps to reproduce:
> 
> echo 3000000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 
> Now all the CPU cores are stuck below 2.2GHz.
> 
> This is after `modprobe amd-pstate replace=1`
> 
> I really don't understand what else I can do. This is reproducible with a
> single command on my Ryzen 7 5800X CPU.

Please drop the replace parameter when loading driver, that patch is not merged by upstream yet.

(In reply to perry_yuan from comment #22)
> Please drop the replace parameter when loading driver, that patch is not
> merged by upstream yet.

This is reproducible when booting with the amd-pstate driver.

Is this driver (development) dead now that there's a new AMD driver for CPPC?

In kernel 6.3.12 limiting CPU frequency doesn't work at all:

echo 3000000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

Average across all cores and maximum frequencies that I observe now are: 3726MHz and 4898MHz.

amd_pstate_highest_perf is readonly, I cannot write to it.

/sys/devices/system/cpu/cpufreq/policy0 $ grep -r . 
energy_performance_available_preferences:default performance balance_performance balance_power power 
amd_pstate_highest_perf:166
scaling_min_freq:550000
scaling_available_governors:performance powersave
scaling_governor:powersave
cpuinfo_max_freq:4929000
amd_pstate_lowest_nonlinear_freq:1752000
amd_pstate_max_freq:4929000
related_cpus:0
scaling_cur_freq:550000
scaling_setspeed:<unsupported>
affected_cpus:0
scaling_max_freq:3000000
cpuinfo_transition_latency:0
energy_performance_preference:performance
scaling_driver:amd-pstate-epp
cpuinfo_min_freq:550000

I've even tried this:

echo 600000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

Zero effect. scaling_max_freq is ignored.

When you're using active mode (EPP), the hardware is choosing frequencies and software only provides a hint to set bias.

You can read a little bit more about active mode at https://www.kernel.org/doc/html/latest/admin-guide/pm/amd-pstate.html

What you should be doing isn't trying to set specific frequency targets, but rather you should change the values of 'energy_performance_preference' to meet the needs for your workload.

(In reply to Mario Limonciello (AMD) from comment #28)
> When you're using active mode (EPP), the hardware is choosing frequencies
> and software only provides a hint to set bias.
> 
> You can read a little bit more about active mode at
> https://www.kernel.org/doc/html/latest/admin-guide/pm/amd-pstate.html
> 
> What you should be doing isn't trying to set specific frequency targets, but
> rather you should change the values of 'energy_performance_preference' to
> meet the needs for your workload.

Thanks Mario.

Would be great if the documentation was updated in this case and the variable removed altogether if the amd-pstate driver is active. I know it's possible.

It's a little bit confusing though, as in Windows I can perfectly set minimum and maximum CPU frequencies and I totally expected this driver to actually allow that.

Now you're saying I need to work with "energy_performance_preference" which doesn't even look like something I can properly configure.

Lastly the documentation mentions: energy_performance_available_preferences, the file that I don't have.

In short it looks like everything is in a deep state of flux.

Lastly there's no boost configuration support which is available both for acpi-cpufreq and Intel's native CPPC driver.

I'll retest the drive in a year. I don't like anything about how it works, functions and its configuration options. It's worse than acpi-cpufreq from seemingly any PoV. Maybe there are some advantages for APUs but on my desktop I only see major disadvantages.

> It's a little bit confusing though, as in Windows I can perfectly set minimum
> and maximum CPU frequencies and I totally expected this driver to actually
> allow that.

I suppose I'm confused as well.  Windows should be using EPP as well, and various software constructs like the power slider will be changing that bias.

Maybe you can better explain exactly what you do in Windows and exactly what is happening.

> Lastly the documentation mentions: energy_performance_available_preferences,
> the file that I don't have.

Eh?  Didn't your above comment show this:

/sys/devices/system/cpu/cpufreq/policy0 $ grep -r . 
energy_performance_available_preferences:default performance balance_performance balance_power power 

> Lastly there's no boost configuration support which is available both for
> acpi-cpufreq and Intel's native CPPC driver.

Something seems quite wrong if you're not getting boost support.
It registers for boost, see https://github.com/torvalds/linux/blob/v6.4/drivers/cpufreq/amd-pstate.c#L1121

> It's worse than acpi-cpufreq from seemingly any PoV. Maybe there are some
> advantages for APUs but on my desktop I only see major disadvantages

I /suspect/ what you'll want to do is change the scaling_governor from 'powersave' to 'performance' for your desktop use case.

Created attachment 304797 [details]
Power Options Processor Power Management in Windows.png

(In reply to Mario Limonciello (AMD) from comment #30)
> > It's a little bit confusing though, as in Windows I can perfectly set
> minimum
> > and maximum CPU frequencies and I totally expected this driver to actually
> > allow that.
> 
> I suppose I'm confused as well.  Windows should be using EPP as well, and
> various software constructs like the power slider will be changing that bias.
> 
> Maybe you can better explain exactly what you do in Windows and exactly what
> is happening.

In Windows I'm free to set minimum and maximum CPU clock speeds to my liking.

Why is it necessary?

1) For instance you want to do benchmarking at certain clock speeds which are not known to be hitting TDP/TTP/temperature limits, thus they show the true potential of your CPU.

2) For instance you want to make you CPU run cooler with a very precise control.

> 
> > Lastly the documentation mentions:
> energy_performance_available_preferences,
> > the file that I don't have.
> 
> Eh?  Didn't your above comment show this:
> 
> /sys/devices/system/cpu/cpufreq/policy0 $ grep -r . 
> energy_performance_available_preferences:default performance
> balance_performance balance_power power 
> 
> > Lastly there's no boost configuration support which is available both for
> > acpi-cpufreq and Intel's native CPPC driver.
> 
> Something seems quite wrong if you're not getting boost support.
> It registers for boost, see
> https://github.com/torvalds/linux/blob/v6.4/drivers/cpufreq/amd-pstate.
> c#L1121

Will recheck in 6.4.x soon.

> 
> > It's worse than acpi-cpufreq from seemingly any PoV. Maybe there are some
> > advantages for APUs but on my desktop I only see major disadvantages
> 
> I /suspect/ what you'll want to do is change the scaling_governor from
> 'powersave' to 'performance' for your desktop use case.

These two options are very coarse vs. being simply able to set maximum clock speeds.

In kernel 6.5 the values of /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq are simply ignored.

Looks like setting the maximum operating CPU frequency is no longer supported.

*** This bug has been marked as a duplicate of bug 217931 ***