Bug 196757 - Turbo Boost can't be disabled with no_turbo
Summary: Turbo Boost can't be disabled with no_turbo
Status: CLOSED DOCUMENTED
Alias: None
Product: Power Management
Classification: Unclassified
Component: intel_pstate (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Srinivas Pandruvada
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-08-24 11:44 UTC by kay-b
Modified: 2017-10-10 07:25 UTC (History)
2 users (show)

See Also:
Kernel Version: 4.12.8-2 (and previous ones)
Subsystem:
Regression: No
Bisected commit-id:


Attachments
dmesg with intel_pstate debug (54.17 KB, text/plain)
2017-08-24 20:22 UTC, kay-b
Details
dmesg with intel_pstate debug and no_turbo=1 (54.94 KB, text/plain)
2017-09-13 12:53 UTC, kay-b
Details
dmesg_after_booted_with_dyndbg_and_then_echoed1_into_no-turbo (54.82 KB, text/plain)
2017-09-20 18:13 UTC, kay-b
Details
turbostat_after_booted_with_dyndbg_and_then_echoed1_into_no-turbo_and-ran-vlc (26.18 KB, text/plain)
2017-09-20 18:14 UTC, kay-b
Details
intel_pstate_tracer_after_echoed1_into_noturbo (720.93 KB, application/x-bzip)
2017-09-20 19:59 UTC, kay-b
Details
intel_pstate_tracer_after_echoed1_into_noturbo_tar-gz (834.30 KB, application/gzip)
2017-09-20 20:18 UTC, kay-b
Details
stress -c 4, no_turbo=1 (564.00 KB, application/x-tar)
2017-09-25 16:35 UTC, kay-b
Details
stress -c 4, no_turbo=1 max_perf_pct=50 (635.49 KB, application/x-tar)
2017-09-25 16:36 UTC, kay-b
Details
"vlc a" and later "stress -c 4", no_turbo=1, max_perf_pct=50 (927.11 KB, application/x-tar)
2017-09-25 16:37 UTC, kay-b
Details
"/sys/kernel/debug/dri/0/i915_ring_freq_table" and "/sys/class/drm/card0/gt*" (287 bytes, application/x-xz)
2017-09-26 10:08 UTC, kay-b
Details
Script to set maximum GPU frequency to match maximum CPU frequncy with no_turbo. (1.39 KB, application/x-shellscript)
2017-09-26 19:52 UTC, kay-b
Details

Description kay-b 2017-08-24 11:44:21 UTC
I have always had this issue on my Lenovo T430 with Intel i5 3320M since purchased approximately a year ago.

When I echo "1" into /sys/devices/system/cpu/intel_pstate/no_turbo, Turbo Boost is still active (under load, of course).

> $ echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
> $ cat /sys/devices/system/cpu/intel_pstate/no_turbo 
> 1

Not every high CPU load does lead to Turbo Boost (which is at least 3.1 GHz on 3320M). For example, this does not:
> $ while true; do true; done         # result: 2.6GHz

But I found it can reliably be reproduced with a known bug in vlc (Videolan Media Player):

> $ vlc vlc://nop --repeat             # result: 3.1GHz
> $ vlc --version
> VLC media player 2.2.6 Umbrella (revision 2.2.6-0-g1aae78981c)

Please tell me, if I shall provide any further information.

Regards
Kay
Comment 1 Srinivas Pandruvada 2017-08-24 19:36:24 UTC
Can you boot by adding to kernel command line

dyndbg="file intel_pstate.c +p" 

Attach dmesg.
Comment 2 kay-b 2017-08-24 20:22:28 UTC
Created attachment 258085 [details]
dmesg with intel_pstate debug

The output of dmesg.

Regards
Kay
Comment 3 kay-b 2017-08-24 20:24:08 UTC
Wait. Having postet it, this actually looks like I entered the parameter wrong... Let me retry if its really wrong. Sorry.
Comment 4 kay-b 2017-08-24 20:29:22 UTC
Okay, same result. I was confused by the
> "dyndbg=file intel_pstate.c +p"
in the second line. Which appears even though I added
> dyndbg="file intel_pstate.c +p"
to linux cmd-line.
Comment 5 Srinivas Pandruvada 2017-09-11 18:11:43 UTC
Does this log contains the log after echo 1 > no_turbo?

I don't see that.
Comment 6 kay-b 2017-09-13 12:53:54 UTC
Created attachment 258357 [details]
dmesg with intel_pstate debug and no_turbo=1
Comment 7 kay-b 2017-09-13 12:54:23 UTC
No, it doesn't - I did not know I had to do that. I'm surprised, that this information is preserved over a reboot. However, I made a new log, see attachment.
Comment 8 Srinivas Pandruvada 2017-09-20 16:19:52 UTC
I don't see log like this when you switch no_turbo to 1
 "[    1.287650] intel_pstate: set_policy cpuinfo.max ... policy->max ..

no_turbo information is not preserved over boot. If it is means that turbo is disabled by the BIOS.

What is dump of the following MSRs without updating the no_turbo?

#rdmsr 0x1a0
#rdmsr 0xce
#rdmsr 0x1ad
#rdmsr 0x648
#rdmsr 0x649
#rdmsr 0x64a
#rdmsr 0x64b
#rdmsr 0x64c

Also run turbostat for the above two scenarios when you disabled turbo and see turbo.
Comment 9 Srinivas Pandruvada 2017-09-20 16:20:14 UTC
Forgot to add
#rdmsr 0x1ad
Comment 10 kay-b 2017-09-20 17:01:53 UTC
For the second log, I echoed "1" into no_turbo, added the command line options you told me to and rebooted. I don't have the link available right now, but I know of at least one other person who posted about this on reddit, who has the same problem.


> cat /sys/devices/system/cpu/intel_pstate/no_turbo 
0
> rdmsr 0x1a0
850089
> rdmsr 0xce
80c10e0011a00
> rdmsr 0x1ad
1f1f1f21
> rdmsr 0x648
1a
> rdmsr 0x649
c0000000000000
> rdmsr 0x64a
c0000000000000
> rdmsr 0x64b
80000000
> rdmsr 0x64c
0
> rdmsr 0x1ad
1f1f1f21



> cat /sys/devices/system/cpu/intel_pstate/no_turbo 
0
> turbostat
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:3a:9 (6:58:9)
CPUID(1): SSE3 MONITOR SMX EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, TURBO, DTS, PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu3: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
cpu3: MSR_MISC_PWR_MGMT: 0x00400000 (ENable-EIST_Coordination DISable-EPB DISable-OOB)
RAPL: 1872 sec. Joule Counter Range, at 35 Watts
cpu3: MSR_PLATFORM_INFO: 0x80c10e0011a00
12 * 100.0 = 1200.0 MHz max efficiency frequency
26 * 100.0 = 2600.0 MHz base frequency
cpu3: MSR_IA32_POWER_CTL: 0x0014005f (C1E auto-promotion: ENabled)
cpu3: MSR_TURBO_RATIO_LIMIT: 0x1f1f1f21
31 * 100.0 = 3100.0 MHz max turbo 4 active cores
31 * 100.0 = 3100.0 MHz max turbo 3 active cores
31 * 100.0 = 3100.0 MHz max turbo 2 active cores
33 * 100.0 = 3300.0 MHz max turbo 1 active cores
cpu3: MSR_CONFIG_TDP_NOMINAL: 0x0000001a (base_ratio=26)
cpu3: MSR_CONFIG_TDP_LEVEL_1: 0xc0000000000000 (PKG_MIN_PWR_LVL1=192 PKG_MAX_PWR_LVL1=0 LVL1_RATIO=0 PKG_TDP_LVL1=0)
cpu3: MSR_CONFIG_TDP_LEVEL_2: 0xc0000000000000 (PKG_MIN_PWR_LVL2=192 PKG_MAX_PWR_LVL2=0 LVL2_RATIO=0 PKG_TDP_LVL2=0)
cpu3: MSR_CONFIG_TDP_CONTROL: 0x80000000 ( lock=1)
cpu3: MSR_TURBO_ACTIVATION_RATIO: 0x00000000 (MAX_NON_TURBO_RATIO=0 lock=0)
cpu3: MSR_PKG_CST_CONFIG_CONTROL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu3: POLL: CPUIDLE CORE POLL IDLE
cpu3: C1: MWAIT 0x00
cpu3: C1E: MWAIT 0x01
cpu3: C3: MWAIT 0x10
cpu3: C6: MWAIT 0x20
cpu3: C7: MWAIT 0x30
cpu3: cpufreq driver: intel_pstate
cpu3: cpufreq governor: powersave
cpufreq intel_pstate no_turbo: 0
cpu3: MSR_MISC_FEATURE_CONTROL: 0x00000000 (L2-Prefetch L2-Prefetch-pair L1-Prefetch L1-IP-Prefetch)
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10000000c00118 (35 W TDP, RAPL 24 - 0 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x8000815e00208118 (locked)
cpu0: PKG Limit #1: ENabled (35.000000 Watts, 64.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (43.750000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x03691200 (105 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88370000 (50 C)
cpu0: MSR_IA32_PACKAGE_THERM_INTERRUPT: 0x00000003 (105 C, 105 C)
cpu3: MSR_PKGC3_IRTL: 0x0000883b (valid, 60416 ns)
cpu3: MSR_PKGC6_IRTL: 0x00008850 (valid, 81920 ns)
cpu3: MSR_PKGC7_IRTL: 0x00008857 (valid, 89088 ns)
^C
> echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo 
> turbostat
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:3a:9 (6:58:9)
CPUID(1): SSE3 MONITOR SMX EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, TURBO, DTS, PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
cpu2: MSR_MISC_PWR_MGMT: 0x00400000 (ENable-EIST_Coordination DISable-EPB DISable-OOB)
RAPL: 1872 sec. Joule Counter Range, at 35 Watts
cpu2: MSR_PLATFORM_INFO: 0x80c10e0011a00
12 * 100.0 = 1200.0 MHz max efficiency frequency
26 * 100.0 = 2600.0 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x0014005f (C1E auto-promotion: ENabled)
cpu2: MSR_TURBO_RATIO_LIMIT: 0x1f1f1f21
31 * 100.0 = 3100.0 MHz max turbo 4 active cores
31 * 100.0 = 3100.0 MHz max turbo 3 active cores
31 * 100.0 = 3100.0 MHz max turbo 2 active cores
33 * 100.0 = 3300.0 MHz max turbo 1 active cores
cpu2: MSR_CONFIG_TDP_NOMINAL: 0x0000001a (base_ratio=26)
cpu2: MSR_CONFIG_TDP_LEVEL_1: 0xc0000000000000 (PKG_MIN_PWR_LVL1=192 PKG_MAX_PWR_LVL1=0 LVL1_RATIO=0 PKG_TDP_LVL1=0)
cpu2: MSR_CONFIG_TDP_LEVEL_2: 0xc0000000000000 (PKG_MIN_PWR_LVL2=192 PKG_MAX_PWR_LVL2=0 LVL2_RATIO=0 PKG_TDP_LVL2=0)
cpu2: MSR_CONFIG_TDP_CONTROL: 0x80000000 ( lock=1)
cpu2: MSR_TURBO_ACTIVATION_RATIO: 0x00000000 (MAX_NON_TURBO_RATIO=0 lock=0)
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C1E: MWAIT 0x01
cpu2: C3: MWAIT 0x10
cpu2: C6: MWAIT 0x20
cpu2: C7: MWAIT 0x30
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: powersave
cpufreq intel_pstate no_turbo: 1
cpu2: MSR_MISC_FEATURE_CONTROL: 0x00000000 (L2-Prefetch L2-Prefetch-pair L1-Prefetch L1-IP-Prefetch)
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10000000c00118 (35 W TDP, RAPL 24 - 0 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x8000815e00208118 (locked)
cpu0: PKG Limit #1: ENabled (35.000000 Watts, 64.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (43.750000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x03691200 (105 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88370000 (50 C)
cpu0: MSR_IA32_PACKAGE_THERM_INTERRUPT: 0x00000003 (105 C, 105 C)
cpu2: MSR_PKGC3_IRTL: 0x0000883b (valid, 60416 ns)
cpu2: MSR_PKGC6_IRTL: 0x00008850 (valid, 81920 ns)
cpu2: MSR_PKGC7_IRTL: 0x00008857 (valid, 89088 ns)
^C
> cat /sys/devices/system/cpu/intel_pstate/no_turbo 
1
Comment 11 Srinivas Pandruvada 2017-09-20 17:39:52 UTC
Thanks. You don't have turbo disabled by BIOS.

I want to the 
"[    1.287650] intel_pstate: set_policy cpuinfo.max ... policy->max ..
when you enter "echo 1 > no_turbo"

So boot with dyn_debug as above. Then do
echo 1 > no_turbo

Then take dmesg without booting the system.

Then run your load and turbostat for a minute and attach output.
Comment 12 kay-b 2017-09-20 18:13:15 UTC
Created attachment 258533 [details]
dmesg_after_booted_with_dyndbg_and_then_echoed1_into_no-turbo
Comment 13 kay-b 2017-09-20 18:14:25 UTC
Created attachment 258535 [details]
turbostat_after_booted_with_dyndbg_and_then_echoed1_into_no-turbo_and-ran-vlc
Comment 14 kay-b 2017-09-20 18:18:30 UTC
As far as I can see in the output of turbostat, the frequency some times is reported to be a bit above 2600000 MHz (the nominal frequency of my cpu). The output is updated very slow.

With

> watch -t -n.1 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq'

and also the cpufreq-plugin for my taskbar, I saw some higher frequencies (above 3GHz).
Comment 15 kay-b 2017-09-20 18:20:24 UTC
I can't disable turbo in my BIOS. The settings are pretty minimal, as if Lenovo was thinking all customers where too stupid to handle some options. ;)

Or, to be more precisely, I think I have the option to disable all CPU power managment which makes my CPU run on 1.2GHz all the time, which is of course not desired.
Comment 16 Srinivas Pandruvada 2017-09-20 19:17:38 UTC
I see that on few occasions we exceeded base frequency. But need to check if we ever requested.

Can you run this tool? This is in upstream kernel tree in 

 tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py

You can set no_turbo to 1 and then run something like this:

#sudo ./intel_pstate_tracer.py -i 30 --name result

It will create result folder and there will be graphs and also trace files.
You can also see the request and results.
Comment 17 kay-b 2017-09-20 19:59:04 UTC
Created attachment 258539 [details]
intel_pstate_tracer_after_echoed1_into_noturbo
Comment 18 kay-b 2017-09-20 20:00:46 UTC
The script looks unsuspicious and is very short, so I gave it a try. As far as I understand, the turbo might have been requested, for example in line 908 of the file cpu.csv (just to help looking into the problem).

Regards
Comment 19 Srinivas Pandruvada 2017-09-20 20:07:47 UTC
Looks like the file is corrupted. Can you attach the zip of whole results folder.
Comment 20 kay-b 2017-09-20 20:18:19 UTC
Created attachment 258541 [details]
intel_pstate_tracer_after_echoed1_into_noturbo_tar-gz

Sorry, something went wrong. I made a tar.gz. Hope thats okay. Extract into current directory with:
tar xf results.tar.gz
Comment 21 kay-b 2017-09-20 20:19:09 UTC
...which is not of the folder but of all files.
Comment 22 Srinivas Pandruvada 2017-09-20 20:43:32 UTC
If you look at the cpu.csv, the driver never requested a state > 26 ("to" column).

But frequency is sometimes more than 2.6GHz. Very interesting !

Instead of vlc can you try to reproduce with non graphics workload

stress -c 4


Also do one more experiment, cap the max_perf_pct to 50% and repeat the stress workload.
Comment 23 kay-b 2017-09-25 16:35:44 UTC
Created attachment 258583 [details]
stress -c 4, no_turbo=1
Comment 24 kay-b 2017-09-25 16:36:21 UTC
Created attachment 258585 [details]
stress -c 4, no_turbo=1 max_perf_pct=50
Comment 25 kay-b 2017-09-25 16:37:33 UTC
Created attachment 258587 [details]
"vlc a" and later "stress -c 4", no_turbo=1, max_perf_pct=50
Comment 26 kay-b 2017-09-25 16:40:33 UTC
Obviously with stress -c 4 only, things do not escalate that much. However, when using "vlc a" again with max_perf_pct=50, I have also seen more than 3GHz again. In the same test, I decided to also start "stress -c 4" after roughly half of the time. Maybe this is interesting.
Comment 27 Srinivas Pandruvada 2017-09-25 17:40:07 UTC
I need to take one more look at logs but from logs it seems that when
- CPU bound workload then no_turbo is  always followed.
- For media/graphics bound workloads the frequency is overridden by GPU requests so you will still see more than turbo.
Comment 28 kay-b 2017-09-25 18:24:45 UTC
Okay, I'm looking forward for any solution. Thank you for the efforts so far.

I can confirm your observations. By the way: The first time I saw this, was when doing heavy computing in the background (video encoding) and starting to watch a video online - so this is relevant for others too, that are interested in not frying their CPU.

Regards
Comment 29 Srinivas Pandruvada 2017-09-25 22:08:54 UTC
Try this
# cat /sys/kernel/debug/dri/0/i915_ring_freq_table
Attach the output of this

Then note the GPU freq corresponding to 2.6GHz. Let's day this is X

# cat /sys/class/drm/card0/gt*
Attach the output

# echo X > gt_max_freq_mzh
# echo X > gt_boost_freq_mhz

Then run vlc with turbo disable and check.
Comment 30 kay-b 2017-09-26 10:08:33 UTC
Created attachment 258609 [details]
"/sys/kernel/debug/dri/0/i915_ring_freq_table" and "/sys/class/drm/card0/gt*"
Comment 31 kay-b 2017-09-26 10:29:02 UTC
It really did work, I'm super happy! Thanks a lot! Finally I can play some simple games without having to detach my laptops internal keyboard and point an extra fan to the CPU cooler. ;)

Might there be any fix to this in intel_pstate, I mean is that possible? Because if not, I will make my own workaround and also post it to the webboard where I first asked for a solution, for others. If a fix is possible, I won't do the latter.

Regards
Comment 32 Srinivas Pandruvada 2017-09-26 16:42:35 UTC
We can't fix intel_pstate for this. I will document this behavior in our intel_pstate documentation.

Can you write a script to do this? This way others can use.
The steps
- write no_turbo for intel_pstate
- Read 
/sys/devices/system/cpu/cpufreq/policy0/scaling_max_freq
- Read
/sys/kernel/debug/dri/0/i915_ring_freq_table
- Find the GT frequency corresponding to the scaling_max_freq read in previous step
- Write the GT frequency to
/sys/class/drm/card0/gt_gt_max_freq_mhz and gt_boost_freq_mhz
(Alternatively you can use intel_gpu_frequency tool to set max, which are part of gpu tools)
Comment 33 kay-b 2017-09-26 19:52:06 UTC
Created attachment 258611 [details]
Script to set maximum GPU frequency to match maximum CPU frequncy with no_turbo.
Comment 34 kay-b 2017-09-26 19:55:01 UTC
I already planned to do so. Thank you for your recommendations on the way of how to do that and also for the support!
Comment 35 kay-b 2017-09-26 19:58:28 UTC
Maybe you can provide some information, on with which other CPUs this might be useful? To be sure, I just wrote it works for Thinkpad T430. But as you can see it can handle frequencies, too. As long as both path and layout of those files is more or less always the same.
Comment 36 Srinivas Pandruvada 2017-09-26 20:35:37 UTC
It is not laptop model specific can be useful for laptops which has GPU demanding more power to meet its max frequency.
Since we are using a debugfs entry the format may change, but didn't change for a long, so I don't think it is concern right now.

Thanks for the script.

Note You need to log in before you can comment on or make changes to this bug.