Bug 218759

Summary: 6.9-rc kernels - with Ryzen 7840HS CPU single core never boosts to max frequency
Product: Power Management Reporter: Gaha Bana (gahabana)
Component: cpufreqAssignee: Mario Limonciello (AMD) (mario.limonciello)
Status: RESOLVED PATCH_ALREADY_AVAILABLE    
Severity: normal CC: gahabana, jlp.bugs, L.Bonnaud, mario.limonciello, mlewis, Perry.Yuan, PisonJay, regressions, thongpv87
Priority: P3    
Hardware: AMD   
OS: Linux   
Kernel Version: 6.9-rc5 Subsystem:
Regression: No Bisected commit-id:
Attachments: cppc check script
Output of CPCC script (on 6.8.0-rc5 kernel)
result of 'git am --show-current-patch=diff'

Description Gaha Bana 2024-04-22 06:50:30 UTC
There is a change from 6.8 kernel to 6.9-rc which causes AMD Ryzen 7840HS CPU not to be able to boost single core CPU.
Geekbench single core score is approx 15% lower (2265 vs 2538). The problem does not show with kernel 6.8 or 6.7.
It is irrelevant whether kernel command line has amd-pstate-epp driver loaded as active or passive or guide.

Simple loading of single core with:
cat /dev/zero > /dev/null

Shows both in 'htop' as well as with 'turbostat' that in the case of kernel 4.9-rc5 (as well as previous rcX) that CPU boosts single core only to 4.35GHz where with previous kernels it goes as high as 5.1GHz.

Here are the outputs of lscpu, and turbopower while running 'cat /dev/zero > /dev/null' 1st with 6.8 kernel (working ok) and than with 6.9-rc5 kernel which has a problem

Hint if it helps - lscpu already shows maximum frequency as 4350Mhz with 6.9-rc kernels

-----  
zh@muc:~$ uname -a
Linux muc 6.8.7-060807-generic #202404170934 SMP PREEMPT_DYNAMIC Wed Apr 17 09:46:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
zh@muc:~$ sudo lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   16
  On-line CPU(s) list:    0-15
Vendor ID:                AuthenticAMD
  BIOS Vendor ID:         Advanced Micro Devices, Inc.
  Model name:             AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
    BIOS Model name:      AMD Ryzen 7 7840HS w/ Radeon 780M Graphics      Unknown CPU @ 3.8GHz
    BIOS CPU family:      107
    CPU family:           25
    Model:                116
    Thread(s) per core:   2
    Core(s) per socket:   8
    Socket(s):            1
    Stepping:             1
    Frequency boost:      enabled
    CPU(s) scaling MHz:   32%
    CPU max MHz:          6080.0000
    CPU min MHz:          400.0000
    BogoMIPS:             7586.12
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdp
                          e1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma
                          cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3d
                          nowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba p
                          erfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx sma
                          p avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total c
                          qm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_cl
                          ean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512
                          _vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca flush_l1d
Virtualization features:
  Virtualization:         AMD-V
Caches (sum of all):
  L1d:                    256 KiB (8 instances)
  L1i:                    256 KiB (8 instances)
  L2:                     8 MiB (8 instances)
  L3:                     16 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-15
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Vulnerable: Safe RET, no microcode
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

zh@muc:~/git/kernel/tools/power/x86/turbostat$ sudo ./turbostat
turbostat version 2024.04.08 - Len Brown <lenb@kernel.org>
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.7-060807-generic root=UUID=dc0bb664-2e43-421d-bd28-96d0332f030c ro amd_pstate=guided iommu=pt quiet
CPUID(0): AuthenticAMD 0x10 CPUID levels
CPUID(1): family:model:stepping 0x19:74:1 (25:116:1) microcode 0x0
CPUID(0x80000000): max_extended_levels: 0x80000028
CPUID(1): SSE3 MONITOR - - - TSC MSR - HT -
CPUID(6): APERF, No-TURBO, No-DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, No-EPB
CPUID(7): No-SGX No-Hybrid
cpu0: cpufreq driver: amd-pstate
cpu0: cpufreq governor: ondemand
cpufreq boost: 1
/dev/cpu_dma_latency: 2000000000 usec (default)
current_driver: acpi_idle
current_governor: menu
current_governor_ro: menu
cpu0: POLL: CPUIDLE CORE POLL IDLE
cpu0: C1: ACPI FFH MWAIT 0x0
cpu0: C2: ACPI IOPORT 0x414
cpu0: C3: ACPI IOPORT 0x415
RAPL: 234 sec. Joule Counter Range, at 280 Watts
cpu0: MSR_RAPL_PWR_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IPC	IRQ	POLL	C1	C2	C3	POLL%	C1%	C2%	C3%	CorWatt	PkgWatt
-	-	481	10.30	4667	3798	1.83	51969	427	4016	12902	10122	0.01	0.31	5.88	83.78	19.52	28.87
0	0	986	22.42	4401	3793	1.37	11392	162	1214	2794	3079	0.09	1.87	14.58	61.81	3.64	28.82
0	8	44	1.33	3351	3793	0.44	1514	0	27	268	721	0.00	0.03	1.87	96.94
1	1	29	1.31	2190	3793	0.51	1802	0	2	204	1059	0.00	0.00	2.72	96.23	0.23
1	9	16	0.62	2531	3793	0.58	478	0	26	107	157	0.00	0.02	0.78	98.62
2	2	1174	26.61	4413	3793	1.37	16684	235	2542	8383	302	0.12	2.76	65.77	5.07	4.96
2	10	225	4.93	4571	3793	1.06	2451	0	47	394	1020	0.00	0.06	2.21	93.01
3	3	15	0.67	2264	3793	0.45	845	0	13	100	402	0.00	0.04	0.80	98.59	0.18
3	11	15	0.67	2302	3793	0.57	542	0	3	60	277	0.00	0.00	0.83	98.56
4	4	14	0.55	2464	3793	0.47	861	2	13	328	286	0.00	0.01	1.74	97.76	0.24
4	12	39	1.43	2736	3793	0.69	871	0	3	65	433	0.00	0.00	0.66	98.01
5	5	2203	44.20	4984	3793	2.12	4691	25	1	24	367	0.02	0.00	0.20	55.63	9.82
5	13	2825	56.38	5011	3793	2.13	7846	3	99	27	1064	0.00	0.05	0.40	43.21
6	6	17	0.60	2796	3793	0.70	517	0	1	31	264	0.00	0.00	0.30	99.16	0.15
6	14	10	0.44	2263	3793	0.44	440	0	0	38	223	0.00	0.00	0.45	99.16
7	7	59	2.42	2453	3793	1.11	862	0	25	78	383	0.00	0.07	0.55	97.05	0.26
7	15	3	0.12	2161	3793	0.62	173	0	0	1	85	0.00	0.00	0.01	99.88

---------------------

zh@muc:~$ uname -a
Linux muc 6.9.0-rc5-zh-250HZ #3 SMP PREEMPT_DYNAMIC Sun Apr 21 22:11:26 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux
zh@muc:~$ sudo lscpu
[sudo] password for zh:
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   16
  On-line CPU(s) list:    0-15
Vendor ID:                AuthenticAMD
  BIOS Vendor ID:         Advanced Micro Devices, Inc.
  Model name:             AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
    BIOS Model name:      AMD Ryzen 7 7840HS w/ Radeon 780M Graphics      Unknown CPU @ 3.8GHz
    BIOS CPU family:      107
    CPU family:           25
    Model:                116
    Thread(s) per core:   2
    Core(s) per socket:   8
    Socket(s):            1
    Stepping:             1
    Frequency boost:      enabled
    CPU(s) scaling MHz:   22%
    CPU max MHz:          4350.0000
    CPU min MHz:          400.0000
    BogoMIPS:             7586.23
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdp
                          e1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor
                          ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misa
                          lignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate
                          ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdse
                          ed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_m
                          bm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_sca
                          le vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku os
                          pke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca flush_l1d amd_lbr
                          _pmc_freeze
Virtualization features:
  Virtualization:         AMD-V
Caches (sum of all):
  L1d:                    256 KiB (8 instances)
  L1i:                    256 KiB (8 instances)
  L2:                     8 MiB (8 instances)
  L3:                     16 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-15
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Vulnerable: No microcode
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
zh@muc:~/git/kernel/tools/power/x86/turbostat$ sudo ./turbostat
turbostat version 2024.04.08 - Len Brown <lenb@kernel.org>
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.9.0-rc5-zh-250HZ root=UUID=dc0bb664-2e43-421d-bd28-96d0332f030c ro amd_pstate=guided iommu=pt quiet
CPUID(0): AuthenticAMD 0x10 CPUID levels
CPUID(1): family:model:stepping 0x19:74:1 (25:116:1) microcode 0x0
CPUID(0x80000000): max_extended_levels: 0x80000028
CPUID(1): SSE3 MONITOR - - - TSC MSR - HT -
CPUID(6): APERF, No-TURBO, No-DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, No-EPB
CPUID(7): No-SGX No-Hybrid
cpu0: cpufreq driver: amd-pstate
cpu0: cpufreq governor: ondemand
cpufreq boost: 1
/dev/cpu_dma_latency: 2000000000 usec (default)
current_driver: acpi_idle
current_governor: menu
current_governor_ro: menu
cpu0: POLL: CPUIDLE CORE POLL IDLE
cpu0: C1: ACPI FFH MWAIT 0x0
cpu0: C2: ACPI IOPORT 0x414
cpu0: C3: ACPI IOPORT 0x415
RAPL: 234 sec. Joule Counter Range, at 280 Watts
cpu0: MSR_RAPL_PWR_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IPC	IRQ	POLL	C1	C2	C3	POLL%	C1%	C2%	C3%	CorWatt	PkgWatt
-	-	292	7.30	4005	3802	2.43	23723	44	656	6574	9594	0.00	0.18	4.10	88.76	4.39	11.77
0	0	52	2.57	2029	3793	0.65	3004	27	175	364	1643	0.00	0.51	3.27	94.01	0.11	11.73
0	8	15	0.75	1991	3793	0.60	946	1	11	212	398	0.00	0.13	1.33	97.89
1	1	37	1.79	2084	3793	1.17	1309	1	26	194	712	0.00	0.25	1.71	96.40	0.60
1	9	645	15.13	4260	3793	2.54	528	0	0	7	140	0.00	0.00	0.04	84.85
2	2	91	3.86	2361	3793	1.02	8182	14	122	4979	2346	0.00	0.18	51.79	44.75	0.37
2	10	13	0.53	2410	3793	0.71	1132	0	21	98	688	0.00	0.14	0.85	98.58
3	3	2243	52.14	4302	3793	2.55	1798	0	30	165	266	0.00	0.56	1.14	46.21	3.02
3	11	1432	33.34	4294	3793	2.56	853	0	8	24	85	0.00	0.10	0.22	66.35
4	4	21	0.98	2191	3793	0.82	820	0	4	66	478	0.00	0.04	0.47	98.61	0.07
4	12	14	0.78	1861	3793	0.88	568	0	8	52	320	0.00	0.00	0.49	98.80
5	5	9	0.47	1932	3793	0.87	416	0	6	34	235	0.00	0.03	0.36	99.17	0.04
5	13	7	0.30	2314	3793	0.67	221	0	10	19	125	0.00	0.03	0.16	99.23
6	6	33	1.62	2024	3793	1.02	1539	0	169	175	796	0.00	0.27	1.92	96.35	0.11
6	14	27	1.40	1896	3793	0.62	1318	1	26	95	764	0.00	0.15	0.81	97.76
7	7	14	0.61	2250	3793	0.72	653	0	25	55	357	0.00	0.35	0.53	98.58	0.05
7	15	10	0.45	2146	3793	0.71	436	0	15	35	241	0.00	0.15	0.23	99.21
Comment 1 Perry Yuan(AMD) 2024-04-22 08:12:46 UTC
Created attachment 306193 [details]
cppc check script
Comment 2 Perry Yuan(AMD) 2024-04-22 08:13:48 UTC
Hi There,
Could you run the script to capture the CPPC capabilities to check the detail?

Perry
Comment 3 Gaha Bana 2024-04-22 10:15:50 UTC
Created attachment 306194 [details]
Output of CPCC script (on 6.8.0-rc5 kernel)

Output of cppc_attr_values_check.sh script.
BTW, it is exactly the same when ran on 'older' 6.8.7 kernel
Comment 4 Gaha Bana 2024-04-24 23:44:24 UTC
Anything else i could do that might help pinpoint or reproduce the issue ?
Comment 5 Perry Yuan(AMD) 2024-04-25 00:39:02 UTC
(In reply to Gaha from comment #4)
> Anything else i could do that might help pinpoint or reproduce the issue ?

Hi Gaha, 

Sorry to reply late,
could you please try the below two patchset ?

1. https://lore.kernel.org/lkml/cover.1713858800.git.perry.yuan@amd.com/
2. https://lore.kernel.org/lkml/cover.1713861200.git.perry.yuan@amd.com/

then switch to amd pstate EPP driver mode, then provide "lscpu -ae" output with "tbench -t 100 2"

then we could see what it changed.

Perry.
Comment 6 Gaha Bana 2024-04-25 10:12:06 UTC
hi, I've applied patches to 6.9.0-rc5 and built kernel with those.
With kernel command line being either "passive", "guided" or "active" result is the same - single core never goes above 4.35Ghz. with 6.8 kernel it does go all the way to 5.1Ghz.

here is the output of lscpu -ae when running Geekbench (during single core load) and it is the same with 'cat /dev/zero > /dev/null' . As for the 'tbench -t 100 2' - on my ubuntu system tbench is part of diskbench. If you were referring to tbench as part of kernel testing suite (https://docs.kernel.org/6.8/admin-guide/pm/amd-pstate.html) - i tried building it, it produces errors as scripts are looking for amd-pstate driver not amd-pstate-epp driver etc...

However, am 100% certain that frequencies never boost above 4.35Ghz.(went back to 6.8.7 kernel and all is good).

Here is the (hopefully useful output) showing that new 'cpp_boost' flag is there etc as well as lspci -ae outputs during single core loading:

zh@muc:~/kselftest/amd-pstate$ cat /sys/devices/system/cpu/amd_pstate/cpb_boost
1
zh@muc:~/kselftest/amd-pstate$ cat /sys/devices/system/cpu/cpufreq/policy0/scaling_driver
amd-pstate-epp

zh@muc:~$ lscpu -ae
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 4350.0000 400.0000  400.0000
  1    0      0    1 1:1:1:0          yes 4350.0000 400.0000 4309.2700
  2    0      0    2 2:2:2:0          yes 4350.0000 400.0000 2416.1770
  3    0      0    3 3:3:3:0          yes 4350.0000 400.0000  400.0000
  4    0      0    4 4:4:4:0          yes 4350.0000 400.0000 1722.2209
  5    0      0    5 5:5:5:0          yes 4350.0000 400.0000  400.0000
  6    0      0    6 6:6:6:0          yes 4350.0000 400.0000  400.0000
  7    0      0    7 7:7:7:0          yes 4350.0000 400.0000  400.0000
  8    0      0    0 0:0:0:0          yes 4350.0000 400.0000 1480.4850
  9    0      0    1 1:1:1:0          yes 4350.0000 400.0000 4307.9858
 10    0      0    2 2:2:2:0          yes 4350.0000 400.0000 1751.6750
 11    0      0    3 3:3:3:0          yes 4350.0000 400.0000 1722.3180
 12    0      0    4 4:4:4:0          yes 4350.0000 400.0000  400.0000
 13    0      0    5 5:5:5:0          yes 4350.0000 400.0000  400.0000
 14    0      0    6 6:6:6:0          yes 4350.0000 400.0000  400.0000
 15    0      0    7 7:7:7:0          yes 4350.0000 400.0000  400.0000

Are there any other tests i could run to help figure out root cause of problem with 6.9 kernel ?
Comment 7 Gaha Bana 2024-04-25 15:21:23 UTC
FYI - sorry if obvious to you & the team - i tried manually compiling 6.8.7 and it works fine. I tried to see if these patches can be applied to it, but that is not the case.
Therefore am certain that whatever is causing scaling in single core workloads not to work it is result of the changes made between 6.8 and 6.9
Appreciate this may not help but here it is.

thank you
Comment 8 Mario Limonciello (AMD) 2024-04-25 16:21:04 UTC
Can you please try with preferred core disabled?

amd_prefcore=disable
Comment 9 Gaha Bana 2024-04-25 18:15:20 UTC
hi , thank you for the suggestions (@Mario), 
    I've just tried that as well. Single core workloads still do not go above 'basic' speeds. In addition to adding 'amd_prefcore=disable' below i've also tried adding explicitly to kernel command line amd_pstate various parameters (active, guided, passive) 'GRUB_CMDLINE_LINUX_DEFAULT="amd_pstate=guided amd_prefcore=disable iommu=pt quiet " with no change in behaviour
zh@muc:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.9.0-rc5-zh-250HZ-amdpatches+ root=UUID=dc0bb664-2e43-421d-bd28-96d0332f030c ro amd_prefcore=disable iommu=pt quiet
zh@muc:~$ uname -a
Linux muc 6.9.0-rc5-zh-250HZ-amdpatches+ #5 SMP PREEMPT_DYNAMIC Thu Apr 25 11:45:21 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux
... (In 2nd session am running 'cat /dev/zero >/dev/null')
zh@muc:~$ lscpu -ae
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 4350.0000 400.0000 1815.3290
  1    0      0    1 1:1:1:0          yes 4350.0000 400.0000 1724.5240
  2    0      0    2 2:2:2:0          yes 4350.0000 400.0000 2156.9199
  3    0      0    3 3:3:3:0          yes 4350.0000 400.0000  400.0000
  4    0      0    4 4:4:4:0          yes 4350.0000 400.0000  400.0000
  5    0      0    5 5:5:5:0          yes 4350.0000 400.0000 2050.2410
  6    0      0    6 6:6:6:0          yes 4350.0000 400.0000  400.0000
  7    0      0    7 7:7:7:0          yes 4350.0000 400.0000  400.0000
  8    0      0    0 0:0:0:0          yes 4350.0000 400.0000  400.0000
  9    0      0    1 1:1:1:0          yes 4350.0000 400.0000  400.0000
 10    0      0    2 2:2:2:0          yes 4350.0000 400.0000  400.0000
 11    0      0    3 3:3:3:0          yes 4350.0000 400.0000  400.0000
 12    0      0    4 4:4:4:0          yes 4350.0000 400.0000  400.0000
 13    0      0    5 5:5:5:0          yes 4350.0000 400.0000  400.0000
 14    0      0    6 6:6:6:0          yes 4350.0000 400.0000 4315.7319
 15    0      0    7 7:7:7:0          yes 4350.0000 400.0000  400.0000

I hope this helps ?
Comment 10 Mario Limonciello (AMD) 2024-04-25 18:49:47 UTC
In that case, would you be able to bisect perhaps?  I think you can *probably* focus the bisect on drivers/cpufreq directory so it should be relatively quick.  

Even if that is indeterminate you'll at least have a bunch of function hashes you can use to reduce the number of steps for a full kernel tree bisect.
Comment 11 Gaha Bana 2024-04-26 00:06:01 UTC
(In reply to Mario Limonciello (AMD) from comment #10)
> In that case, would you be able to bisect perhaps?  I think you can
> *probably* focus the bisect on drivers/cpufreq directory so it should be
> relatively quick.  
> 
> Even if that is indeterminate you'll at least have a bunch of function
> hashes you can use to reduce the number of steps for a full kernel tree
> bisect.

Hi, tx but am not sure I'd know how to do that. 
Just wanted to report a bug and help to the best of my abilities to get it sorted out as it may impact more people down the road.
Comment 12 Mario Limonciello (AMD) 2024-04-26 00:10:59 UTC
This should help you get started on a bisect: https://www.kernel.org/doc/html/latest/admin-guide/bug-bisect.html

If you can build a kernel you basically take a good and bash hash and then pick a point in between and keep going at half way points until you found the broken commit.
Comment 13 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-04-26 08:18:38 UTC
FWIW, a more verbose bisection guide can be found here:
https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html

While at it, let me add this to the regression tracking:

#regzbot introduced: v6.8..v6.9-rc5
#regzbot summary: Ryzen 7840HS CPU single core never boosts to max frequency
Comment 14 Perry Yuan(AMD) 2024-04-26 08:28:31 UTC
Hi all,

I have got issue root cause and working on new fix patch, there is one power firmware change on this generation CPU.

will share the testing patch once I got it done.


Perry.
Comment 15 Gaha Bana 2024-04-26 13:59:16 UTC
:thumbs-up: :)
Comment 16 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-05-03 08:49:41 UTC
(In reply to Perry Yuan(AMD) from comment #14)
> I have got issue root cause and working on new fix patch, 

What's the status? 6.9 might be released in 9 days and it would be good to have this fixed.
Comment 17 Mario Limonciello (AMD) 2024-05-03 11:08:41 UTC
Perry is on leave right now for Labor Day so I'll update the status.

The fix is developed but under internal code review at the moment. I don't expect it to make 6.9 but it's small enough we can aim for it to go to 6.9.y.
Comment 18 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-05-03 12:24:59 UTC
thx for the update, Mario!
Comment 19 Perry Yuan(AMD) 2024-05-07 08:41:10 UTC
Hi, Please try this patchset for the issue

https://lore.kernel.org/lkml/cover.1715065568.git.perry.yuan@amd.com/
Comment 20 Gaha Bana 2024-05-07 09:13:56 UTC
hi, 
   i tried applying the batch to linux rcX tree and i can not get it to apply the patch. Have tried v6.9-rc7, -rc6 , -rc5 and -rc1 - all fail with the same msg:

zh@muc:~/git/linus-kernel$ git am ./20240507_perry_yuan_amd_pstate_driver_fixes_and_improvements.mbx
Applying: cpufreq: amd-pstate: optimiza the initial frequency values verification
error: patch failed: drivers/cpufreq/amd-pstate.c:873
error: drivers/cpufreq/amd-pstate.c: patch does not apply
Patch failed at 0001 cpufreq: amd-pstate: optimiza the initial frequency values verification
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

should I apply this to a different 'base' kernel code ?

Will post file as a result of 'git am --show-current-patch=diff' as well if it helps:
Comment 21 Gaha Bana 2024-05-07 09:14:57 UTC
Created attachment 306267 [details]
result of 'git am --show-current-patch=diff'
Comment 22 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-05-07 10:29:11 UTC
(In reply to Perry Yuan(AMD) from comment #19)
> https://lore.kernel.org/lkml/cover.1715065568.git.perry.yuan@amd.com/

Correct me if I'm wrong, but from Mario's earlier comment, the number of patches, and number of changed lines this does sounds like this won't be applied for 6.9 this late; there are no Fixes or stable tags either.

So what's the plan to prevent this regression from entering 6.9? Do we know which commit caused this? Can we revert that one and reapply it for 6.10 (together with these changes, once reviewed).
Comment 23 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-05-07 10:30:02 UTC
Also: how many users do we suspect will run into this?
Comment 24 Mario Limonciello (AMD) 2024-05-07 12:19:48 UTC
It's one patch in the series that was added to the series that helps, but a lot of this code is intertwined. I'll look whether it can move to front of the series if reordered.
Comment 25 Mario Limonciello (AMD) 2024-05-07 12:21:14 UTC
Try to apply just patch 10 in that above link.
Comment 26 Mario Limonciello (AMD) 2024-05-07 12:22:22 UTC
But that might need patch 9 too, we'll have to see what really makes sense for stable.
Comment 27 Gaha Bana 2024-05-07 13:00:01 UTC
(In reply to Mario Limonciello (AMD) from comment #26)
> But that might need patch 9 too, we'll have to see what really makes sense
> for stable.

Thank you Mario!  I tried applying only 10 or 9&10 (git am -i ./patchfromperry.mbx) and it fails in the same way.

Have even played with previous other pstate patches (latest versions of those mentioned at the top of this thread that Perry mentioned may help) - with no avail.

So my question is please the same:
 *** to which source version these patches correspond to and what are the prerequiste patches (if any) ? ***
Comment 28 Mario Limonciello (AMD) 2024-05-07 13:20:22 UTC
Everything should be based off the bleeding-edge branch for linux-pm which is code headed to 6.10.
Comment 29 Gaha Bana 2024-05-07 13:23:58 UTC
(In reply to Mario Limonciello (AMD) from comment #28)
> Everything should be based off the bleeding-edge branch for linux-pm which
> is code headed to 6.10.

Thank you Mario - not so simple for not-kernel-developers :( Appreciate the response and clarity though :) 

Was hoping to be able to test and report but will need to wait a month or so i guess for 6.10-rc1 to be able to do that. Pity 6.9 will limit single core performance on loads of AMD Ryzen CPUs ... unless Perry manages to address that issue w/o requiring all the other patches ... still not clear which change from 6.8 to 6.9 caused this to go 'wrong'

Thank you all !
Comment 30 Gaha Bana 2024-05-07 13:24:30 UTC
(In reply to Mario Limonciello (AMD) from comment #28)
> Everything should be based off the bleeding-edge branch for linux-pm which
> is code headed to 6.10.

Thank you Mario - not so simple for not-kernel-developers :( Appreciate the response and clarity though :) 

Was hoping to be able to test and report but will need to wait a month or so i guess for 6.10-rc1 to be able to do that. Pity 6.9 will limit single core performance on loads of AMD Ryzen CPUs ... unless Perry manages to address that issue w/o requiring all the other patches ... still not clear which change from 6.8 to 6.9 caused this to go 'wrong'

Thank you all !
Comment 31 Gaha Bana 2024-05-07 17:48:44 UTC
btw i did clone both linux-next as well as linux-pm and could not apply those patches to either of them :(
Comment 32 Mario Limonciello (AMD) 2024-05-07 17:57:00 UTC
I've pulled out just the relevant fix to it's own patch here:

https://lore.kernel.org/linux-pm/20240507174810.46709-1-mario.limonciello@amd.com/T/#u

I applied that to 6.9-rc7 and tested it.  Can you see if that helps you?
Comment 33 Gaha Bana 2024-05-07 18:50:35 UTC
hi Mario - FANTASTIC job - Thank you :) !
Applied to -rc7 - recompiled - all good.
When running single core tasks on my Ryzen 7840HS single core does go to almost 5.1GHz as it used to... Geekbench scores are great.
I tried only with 'amd_pstate=active' as kernel parameter - should i test with 'guided' and/or 'passive' , or it is a waste of time as this fix addressed all scenarios ?
Comment 34 Mario Limonciello (AMD) 2024-05-07 18:57:57 UTC
Great! It should apply equally to all modes.  If there is a regression with any other mode it should be a different root cause.

If you feel comfortable doing so please add a "Tested-by" tag publicly on the link I posted, instructions at bottom of the page.
Comment 35 Gaha Bana 2024-05-07 19:35:53 UTC
(In reply to Mario Limonciello (AMD) from comment #32)
> I've pulled out just the relevant fix to it's own patch here:
> 
> https://lore.kernel.org/linux-pm/20240507174810.46709-1-mario.
> limonciello@amd.com/T/#u
> 
> I applied that to 6.9-rc7 and tested it.  Can you see if that helps you?

Tested-by: Gaha Bana <gahabana@gmail.com>
Comment 36 Gaha Bana 2024-05-07 19:36:52 UTC
(In reply to Mario Limonciello (AMD) from comment #34)
> Great! It should apply equally to all modes.  If there is a regression with
> any other mode it should be a different root cause.
> 
> If you feel comfortable doing so please add a "Tested-by" tag publicly on
> the link I posted, instructions at bottom of the page.

I tried in 'guided' mode and all is still perfect. Hope my response/tested-by is satisfactory - if not please advise on how to do it properly!

Tx again Mario !
Comment 37 Thong Pham 2024-05-15 06:13:06 UTC
(In reply to Mario Limonciello (AMD) from comment #32)
> I've pulled out just the relevant fix to it's own patch here:
> 
> https://lore.kernel.org/linux-pm/20240507174810.46709-1-mario.
> limonciello@amd.com/T/#u
> 
> I applied that to 6.9-rc7 and tested it.  Can you see if that helps you?

In kernel 6.8 and below, I have some cpu cores that can goes above 5.1GHz. I'm not sure if I read the cpu info correctly: https://preview.redd.it/hp-845-g10-7840hs-32-gb-ddr5-cpu-frequency-mystery-v0-oaorgqghrl1c1.png?width=2560&format=png&auto=webp&s=ab7281dbbeeed86a70f5de23a6db85337d2c900e

That's also match the data in /sys/devices/system/cpu/cpu*/cpufreq/amd_pstate_max_freq

If the clock freq reported by the system above is correct, does your patch limit the clock freq of all cpu core at 5.13Ghz?
Comment 38 Thong Pham 2024-05-15 06:14:55 UTC
Here is cpu max freq that I got from /sys/devices/system/cpu/cpu*/cpufreq/amd_pstate_max_freq

https://preview.redd.it/hp-845-g10-7840hs-32-gb-ddr5-cpu-frequency-mystery-v0-tlmlpomdrl1c1.png?width=673&format=png&auto=webp&s=12eb2184e83bba7628bd28c59515b78549a04547
Comment 39 Perry Yuan(AMD) 2024-05-15 07:44:08 UTC
(In reply to Thong Pham from comment #38)
> Here is cpu max freq that I got from
> /sys/devices/system/cpu/cpu*/cpufreq/amd_pstate_max_freq
> 
> https://preview.redd.it/hp-845-g10-7840hs-32-gb-ddr5-cpu-frequency-mystery-
> v0-tlmlpomdrl1c1.
> png?width=673&format=png&auto=webp&s=12eb2184e83bba7628bd28c59515b78549a04547

please run "lscpu -ae" which will tell you the frequency range and current each cpu frequency running.

maybe you need to run some workloads to trigger the frequency level up.

Perry.
Comment 40 Gaha Bana 2024-05-15 08:03:24 UTC
With 6.8.9 kernel values 'reported' are not correct. With the patch provided above against 6.9 kernel values reported are correct.

However with either kernel CPU is boosting correctly (both in single core and multicore workloads). Geekbench scores are same...

I am not sure which bug with 6.8 is causing wrong reporting of the actual frequencies but 7840HS does not boost to 6.080 GHz but rather only to 5.1 (on my system it shows 5.13GHz) which seems accurate reviewing AMD's documentation.

Therefore I think there is a bug when it comes to reporting actual core frequency with 6.8 kernel BUT in spite of that it works 'correctly'. 6.9 kernel w/o the patch does not boost single core to 5.1GHz (stays at 4.3GHz range) but reports frequencies correctly ... with the patch both boost and reporting is accurate.

Here is lscpu -ae on my system :
----- with kernel 6.8.9 (wrong reporting, works fine):
zh@muc:~$ lscpu -ae
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 5137.0000 400.0000 3675.1870
  1    0      0    1 1:1:1:0          yes 6080.0000 400.0000 1958.0250
  2    0      0    2 2:2:2:0          yes 5293.0000 400.0000 4911.4199
  3    0      0    3 3:3:3:0          yes 6080.0000 400.0000 3120.8420
  4    0      0    4 4:4:4:0          yes 5764.0000 400.0000 1962.1050
  5    0      0    5 5:5:5:0          yes 5449.0000 400.0000 2737.4810
  6    0      0    6 6:6:6:0          yes 5924.0000 400.0000 2769.7849
  7    0      0    7 7:7:7:0          yes 5608.0000 400.0000 4908.4858
  8    0      0    0 0:0:0:0          yes 5137.0000 400.0000 3839.7900
  9    0      0    1 1:1:1:0          yes 6080.0000 400.0000 1957.7570
 10    0      0    2 2:2:2:0          yes 5293.0000 400.0000 4905.7222
 11    0      0    3 3:3:3:0          yes 6080.0000 400.0000 2735.6450
 12    0      0    4 4:4:4:0          yes 5764.0000 400.0000 1958.1840
 13    0      0    5 5:5:5:0          yes 5449.0000 400.0000 2908.0100
 14    0      0    6 6:6:6:0          yes 5924.0000 400.0000 2670.1431
 15    0      0    7 7:7:7:0          yes 5608.0000 400.0000 4895.6372

----- with kernel 6.9.0(with patch applied ,reporting correct, works fine):
zh@muc:/sys/devices/system/cpu/amd_pstate$ lscpu -ae
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 5137.0000 400.0000  542.1100
  1    0      0    1 1:1:1:0          yes 5137.0000 400.0000  542.1100
  2    0      0    2 2:2:2:0          yes 5137.0000 400.0000 2388.0220
  3    0      0    3 3:3:3:0          yes 5137.0000 400.0000 1313.9700
  4    0      0    4 4:4:4:0          yes 5137.0000 400.0000 1375.8060
  5    0      0    5 5:5:5:0          yes 5137.0000 400.0000  447.3700
  6    0      0    6 6:6:6:0          yes 5137.0000 400.0000  400.0000
  7    0      0    7 7:7:7:0          yes 5137.0000 400.0000  400.0000
  8    0      0    0 0:0:0:0          yes 5137.0000 400.0000  400.0000
  9    0      0    1 1:1:1:0          yes 5137.0000 400.0000  400.0000
 10    0      0    2 2:2:2:0          yes 5137.0000 400.0000 3594.5710
 11    0      0    3 3:3:3:0          yes 5137.0000 400.0000  778.9600
 12    0      0    4 4:4:4:0          yes 5137.0000 400.0000  400.0000
 13    0      0    5 5:5:5:0          yes 5137.0000 400.0000  400.0000
 14    0      0    6 6:6:6:0          yes 5137.0000 400.0000  400.0000
 15    0      0    7 7:7:7:0          yes 5137.0000 400.0000  400.0000
Comment 41 Perry Yuan(AMD) 2024-05-16 02:40:47 UTC
https://www.amd.com/zh-hans/products/apu/amd-ryzen-7-pro-7840hs

Yes, looks like the fix work for you, the correct max freq is 5.1GHz as the spec. 

Perry.
Comment 42 Gaha Bana 2024-05-18 19:04:39 UTC
(In reply to Perry Yuan(AMD) from comment #41)
> https://www.amd.com/zh-hans/products/apu/amd-ryzen-7-pro-7840hs
> 
> Yes, looks like the fix work for you, the correct max freq is 5.1GHz as the
> spec. 
> 
> Perry.

Hi, Perry, yes - I think with the fix correct/accurate frequencies are shown.
However, as you see - it shows the same max single core frequency for all the cores though in my experimenting some cores boost more then the others. 
With kernel 6.8 though system worked well reporting showed incorrect max frequencies BUT it had correctly guessed max single core frequency for each of the cores. I think with the patch, maximum value is taken for ALL the cores... I don't think it impacts ME but some folks (and maybe some other CPUs) depend on scheduler making the best possible decision on which core to put a thread on and it does that probably based on number of factors o/w one might be - how fast can this core run.

With this patch - all cores look equally good ... so I am not an expert nor familier with intricacies - this patch DOES fix the bug where CPU is not able to boost any of the cores ... so that is fixed and max frequency shown is also accurate :) hope this comment helps, rather then bring more confusion !
Comment 43 Perry Yuan(AMD) 2024-05-20 16:52:45 UTC
(In reply to Gaha Bana from comment #42)
> (In reply to Perry Yuan(AMD) from comment #41)
> > https://www.amd.com/zh-hans/products/apu/amd-ryzen-7-pro-7840hs
> > 
> > Yes, looks like the fix work for you, the correct max freq is 5.1GHz as the
> > spec. 
> > 
> > Perry.
> 
> Hi, Perry, yes - I think with the fix correct/accurate frequencies are shown.
> However, as you see - it shows the same max single core frequency for all
> the cores though in my experimenting some cores boost more then the others. 
> With kernel 6.8 though system worked well reporting showed incorrect max
> frequencies BUT it had correctly guessed max single core frequency for each
> of the cores. I think with the patch, maximum value is taken for ALL the
> cores... I don't think it impacts ME but some folks (and maybe some other
> CPUs) depend on scheduler making the best possible decision on which core to
> put a thread on and it does that probably based on number of factors o/w one
> might be - how fast can this core run.
> 
> With this patch - all cores look equally good ... so I am not an expert nor
> familier with intricacies - this patch DOES fix the bug where CPU is not
> able to boost any of the cores ... so that is fixed and max frequency shown
> is also accurate :) hope this comment helps, rather then bring more
> confusion !

The 5.1GHz is the max frequency each core can reach, even the core can boost frequency sometimes, the max still is 5.1GHz, the different highest perf values of cores are not used for cpufreq boosting, it is used for other design purpose, so the frequency works well now. 

please send test-by flags to the patch set if possible.
thanks.

Perry.
Comment 44 Gaha Bana 2024-05-21 03:50:39 UTC
(In reply to Perry Yuan(AMD) from comment #43)
> (In reply to Gaha Bana from comment #42)
> > (In reply to Perry Yuan(AMD) from comment #41)
> > > https://www.amd.com/zh-hans/products/apu/amd-ryzen-7-pro-7840hs
> > > 
> > > Yes, looks like the fix work for you, the correct max freq is 5.1GHz as
> the
> > > spec. 
> > > 
> > > Perry.
> > 
> > Hi, Perry, yes - I think with the fix correct/accurate frequencies are
> shown.
> > However, as you see - it shows the same max single core frequency for all
> > the cores though in my experimenting some cores boost more then the others. 
> > With kernel 6.8 though system worked well reporting showed incorrect max
> > frequencies BUT it had correctly guessed max single core frequency for each
> > of the cores. I think with the patch, maximum value is taken for ALL the
> > cores... I don't think it impacts ME but some folks (and maybe some other
> > CPUs) depend on scheduler making the best possible decision on which core
> to
> > put a thread on and it does that probably based on number of factors o/w
> one
> > might be - how fast can this core run.
> > 
> > With this patch - all cores look equally good ... so I am not an expert nor
> > familier with intricacies - this patch DOES fix the bug where CPU is not
> > able to boost any of the cores ... so that is fixed and max frequency shown
> > is also accurate :) hope this comment helps, rather then bring more
> > confusion !
> 
> The 5.1GHz is the max frequency each core can reach, even the core can boost
> frequency sometimes, the max still is 5.1GHz, the different highest perf
> values of cores are not used for cpufreq boosting, it is used for other
> design purpose, so the frequency works well now. 
> 
> please send test-by flags to the patch set if possible.
> thanks.
> 
> Perry.

hi Perry, 
    I've already tagged when Mario posted the link to the patch 'Tested-By' at comment #35 - please see : https://bugzilla.kernel.org/show_bug.cgi?id=218759#c35

Let me know if additional action from my side is needed or what precisely you need me to do ? Thank you !
Comment 45 Mario Limonciello (AMD) 2024-05-21 10:32:17 UTC
Yeah it got merged already in 6.10, no concerns now.
Comment 46 Gaha Bana 2024-05-21 10:34:35 UTC
(In reply to Mario Limonciello (AMD) from comment #45)
> Yeah it got merged already in 6.10, no concerns now.

Not sure if important - but saw it was not in 6.9.1 ... would be great to get it into 6.9.2 as it's a regression ... I am ok with custom compiling 6.9.x and applying patch to it but maybe number of linux distros will use 6.9.x and out of the box it won't work for them as well as it should ?
Comment 47 Mario Limonciello (AMD) 2024-05-21 10:38:52 UTC
It's in the stable queue and will be in an upcoming 6.9.y release.
Comment 48 Gaha Bana 2024-05-21 19:58:56 UTC
(In reply to Mario Limonciello (AMD) from comment #47)
> It's in the stable queue and will be in an upcoming 6.9.y release.

Thumbs-up !!! :) Thank you all !!!
Comment 49 penguinbait 2024-08-09 11:02:25 UTC
(base) snoopy@fedora:~$ uname -a
Linux fedora 6.10.3-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug  5 14:30:00 UTC 2024 x86_64 GNU/Linux

On 6.9 / 6.10 kernels all cores run at same max speed 5312Mhz

On 6.8 and prior, it is like this
Cores 0-1 5132Mhz
Cores 2-3 6076Mhz
Cores 4-5 5289Mhz
Cores 6-7 6076Mhz
Cores 8-9 5447Mhz
Cores 10-11 5760Mhz
Cores 12-13 5695Mhz
Cored 14-15 5918Mhz

based on the above I can no longer boost to higher than 5312Mhz and that wont be fixed?  If I am running 6.8, I can run higher clock speeds on some cores as shown above.  Additionally I can pin work to those cores forcing to run at 6076Mhz.  Its been stable without issues.  I have been staying on 6.8 waiting for fix but it seems one won't be coming ?
Comment 50 penguinbait 2024-08-09 11:04:27 UTC
Sorry 5132Mhz, not 5312Mhz
Comment 51 PisonJay 2024-08-19 17:24:04 UTC
It seems Ryzen AI 9 processors also need this patch.
My HX 365 can theoretically boost to 5.0 GHz, but the maximum frequency shown is only 4.32 GHz.
Comment 52 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-08-20 07:05:32 UTC
If anyone still has problems, please open a new ticket that works stand-alone (short references to this one are fine) and afterwards drop a link to it here, as this one has become confusing[1]; when doing so, also mention if things worked in earlier kernel versions -- and please CC me if they did.

[1] https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kernel-bug-reports-are-ignored/#you-reported-the-problem-in-a-reply-to-an-earlier-report