Created attachment 110741 [details] cpuinfo output Since updating to 3.10 or 3.11, I sometimes find my Dell Lattitude be stuck at very low CPU frequency levels despite it is not overheated and powered by the adaptor. This causes desite sluggish user experience e.g. build-scripts which usually take 1:30 to run for over 5:00 minutes. I first thought this is caused by a buggy BIOS version, so I updated to the latest available version A18 (June 2013) - which cured the poweroff-problems at reboot I was experiencing (introduced with 3.11), but didn't help the frequency scaling issues. Syslog doesn't show any indication what is going on, the only message I saw which could be somehow related is: [30440.014649] perf samples too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 I currently do not have time to bisect the issue, but keep it files in case there are others experiencing the same issues.
Created attachment 110751 [details] syslog
Could you try to disable speedstep in the Bios? From bug 59821, the reporter also meet the same issue and fix by the previous change. https://bugzilla.kernel.org/show_bug.cgi?id=59821
Disabling speedstep in BIOS seems to help to some degree, however Turbo doesn't kick in anymore (according to powertop the CPU seems to be stuck at it's base-clock of 2.8ghz) :/ A build of one of my java-projects takes now 2:30 instead of 1:30.
So disable speedstep can resolve lowe CPU frequency issue but Turbo mode doesn't work and cpufreq remain 2.8GHz during heavy workload, right? Please provide the output of "grep . /sys/bus/cpu/devices/cpu*/cpufre/*" and "grep . /sys/devices/system/cpu/cpufreq/*"
Created attachment 112101 [details] /sys/bus/cpu/devices/
Created attachment 112111 [details] sys/devices/system/cpu/
Exactly, disabling speedstep helps, but turbo doesn't kick in anymore. As your grep command didn't work, I simply stacked all the files into archives.
Ping
Hi, could you try the v3.12 kernel. I see some intel_pstate turbo related patches are upstreamed recently. So current issue became intel_pstate doesn't kick turbo and Cc Dirk.
I just tried the 3.12 kernel on a Dell L502x, and this bug is still present: with speed-step disabled, turbostat reports the cores running at 1.99GHz with 'stress --cpu 8' running. With speed-step enabled, the cores run at 2.56GHz (compared to a turbo max of 2.9GHz). There is other weird intel_pstate stuff going on - at one point yesterday when I was running kernel 3.12, the laptop had all cores locked at the min frequency (800MHz) no matter what, and this persisted across eight (8) reboots. (This was all with speed-step enabled.) The only thing that restored frequency scaling was rebooting into the 3.11 kernel. After that, it was fine when I rebooted back into 3.12.
Hi Dirk: Could you take care this bug? Otherwise, one question: speed-step affects intel_pstate? The symptom seems "yes". Intel_pstate use MSR registers and Bios can involve in?
Can you run "grep . *" in /sys/devices/system/cpu/cpu0/cpufreq with speed_step enabled/disabled and post the results. If you see the lock up at low frequency again could you do the same thing. intel_pstate does not save any information across reboots so it feels like the BIOS is getting in the way somehow. Also have you seen these issues after suspend/resume cycles?
Created attachment 118151 [details] output of grep for speedstep enabled/disabled Here are the results of that grep with speedstep enabled and disabled. With speedstep disabled, turbo doesn't kick after a suspend/resume cycle, either. I haven't seen the lock up at low freq again but I'll keep an eye out for it.
I've just seen the lock up at low freq again after a suspend from ram using kernel 3.12.6. turbostat shows each CPU at 0.80 GHz no matter what I run. Here's the output from the grep command: affected_cpus:0 cpuinfo_cur_freq:799921 cpuinfo_max_freq:2900000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:powersave scaling_max_freq:2900000 scaling_min_freq:800000 scaling_setspeed:<unsupported>
Same here, after suspend to ram the system was horribly slow. While the system was affected, running the grep-result yielded an interesting result - instead of beeing stuck at a single low frequency, the frequency oscillated between 612mhz and ~2.2ghz while running "7za b" procuing load on all 4 logical cores: [root@user-pc cpufreq]# grep . * affected_cpus:0 cpuinfo_cur_freq:2200515 cpuinfo_max_freq:3500000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:powersave scaling_max_freq:3500000 scaling_min_freq:800000 scaling_setspeed:<unsupported> [root@user-pc cpufreq]# grep . * affected_cpus:0 cpuinfo_cur_freq:628578 cpuinfo_max_freq:3500000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:powersave scaling_max_freq:3500000 scaling_min_freq:800000 scaling_setspeed:<unsupported> [root@user-pc cpufreq]# grep . * affected_cpus:0 cpuinfo_cur_freq:1170312 cpuinfo_max_freq:3500000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:powersave scaling_max_freq:3500000 scaling_min_freq:800000 scaling_setspeed:<unsupported> [root@user-pc cpufreq]# grep . * affected_cpus:0 cpuinfo_cur_freq:629015 cpuinfo_max_freq:3500000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:powersave scaling_max_freq:3500000 scaling_min_freq:800000 scaling_setspeed:<unsupported> Processing speed is way lower than normal when the machine shows this behaviour: [ce@user-pc ~]$ 7za b 7-Zip (A) [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18 p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs) RAM size: 7865 MB, # CPU hardware threads: 4 RAM usage: 850 MB, # Benchmark threads: 4 Dict Compressing | Decompressing Speed Usage R/U Rating | Speed Usage R/U Rating KB/s % MIPS MIPS | KB/s % MIPS MIPS 22: 3759 291 1254 3657 | 41169 383 970 3714 23: 3607 288 1275 3675 | 34997 363 881 3202 24: 3617 290 1340 3889 | 35905 377 883 3331 25: 3597 292 1408 4107 | 38861 383 953 3654 ---------------------------------------------------------------- Avr: 290 1319 3832 377 922 3475 Tot: 333 1121 3653
Could somebody please set the bug status to "New" or "Assigned", please. As mentioned earlier the problem is still there with 3.12.8, here are the frequency statistics of powertop while running the 7za multithreaded benchmark: Package | Core | CPU 0 CPU 2 | | Actual 1136 MHz 1116 MHz Idle 0.3% | Idle 0.8% | Idle 29.3% 1.3% 1.71 GHz 3.0% | 1.71 GHz 3.0% | 1.71 GHz 2.3% 2.7% 1.60 GHz 5.3% | 1.60 GHz 4.7% | 1.60 GHz 3.1% 4.6% 1500 MHz 1.9% | 1500 MHz 1.9% | 1500 MHz 1.6% 1.9% 1400 MHz 1.9% | 1400 MHz 1.9% | 1400 MHz 1.9% 1.9% 1300 MHz 4.9% | 1300 MHz 4.9% | 1300 MHz 3.9% 4.9% 1100 MHz 1.6% | 1100 MHz 1.6% | 1100 MHz 1.3% 1.6% 1200 MHz 3.2% | 1200 MHz 3.2% | 1200 MHz 1.3% 3.2% 1000 MHz 2.7% | 1000 MHz 2.7% | 1000 MHz 1.3% 2.7% 900 MHz 6.4% | 900 MHz 6.4% | 900 MHz 4.5% 6.4% 800 MHz 33.2% | 800 MHz 33.2% | 800 MHz 22.7% 33.2% 2.81 GHz 9.3% | 2.81 GHz 9.3% | 2.81 GHz 6.5% 9.3% 3.50 GHz 0.7% | 3.50 GHz 0.7% | 3.50 GHz 0.7% 0.7% 2.40 GHz 8.0% | 2.40 GHz 8.0% | 3.10 GHz 0.2% 8.0% 3.10 GHz 0.2% | 3.10 GHz 0.2% | 2.50 GHz 0.2% 0.2% 2.21 GHz 2.4% | 2.21 GHz 2.4% | 1.91 GHz 0.7% 2.4% 2.00 GHz 5.2% | 2.00 GHz 5.2% | 2.40 GHz 4.9% 5.2% 2.71 GHz 0.2% | 2.71 GHz 0.2% | 2.00 GHz 3.5% 0.2% 1.91 GHz 1.2% | 1.91 GHz 1.2% | 2.21 GHz 1.9% 1.2% 2.50 GHz 0.2% | 2.50 GHz 0.2% | 2.10 GHz 1.6% 0.2% 2.10 GHz 1.6% | 2.10 GHz 1.6% | 3.00 GHz 1.2% 1.6% 3.00 GHz 1.2% | 3.00 GHz 1.2% | 2.60 GHz 1.9% 1.2% 2.60 GHz 1.9% | 2.60 GHz 1.9% | 2.31 GHz 0.7% 1.9% 2.31 GHz 0.7% | 2.31 GHz 0.7% | 1.80 GHz 2.6% 0.7% 1.80 GHz 2.6% | 1.80 GHz 2.6% | 2.6
Does this still happen on the current kernel?
There are comments in this bug report about CPU frequencies locking up at some rate, but it isn't clear how this happens. Somewhere and sometime ago, I mentioned one method. I'll mention it again: In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the intel_pstate driver: echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo All CPU's will now be locked at whatever frequency they were at when the above command was issued.
(In reply to Doug Smythies from comment #18) > There are comments in this bug report about CPU frequencies locking up at > some rate, but it isn't clear how this happens. Somewhere and sometime ago, > I mentioned one method. I'll mention it again: > > In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the > intel_pstate driver: > > echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo > > All CPU's will now be locked at whatever frequency they were at when the > above command was issued. I think you mean all cpu' will NOT be locked...? Clemens and rocko, please check if the problem still exists in the latest upstream kernel.
Yes, this is still happening on the latest kernel, 3.15-rc8 (3.15.0-031500rc8-generic from the Ubuntu mainline kernel ppa). Currently my system is sluggish and /proc/cpuinfo indicates that all my cores are stuck at 801.25 MHz. Here's the output from the earlier grep command: affected_cpus:0 cpuinfo_cur_freq:801250 cpuinfo_max_freq:2900000 cpuinfo_min_freq:800000 cpuinfo_transition_latency:4294967295 related_cpus:0 scaling_available_governors:performance powersave scaling_driver:intel_pstate scaling_governor:performance scaling_max_freq:2900000 scaling_min_freq:800000 scaling_setspeed:<unsupported> In case it's relevant, I do have this in the syslog: Jun 3 00:21:10 trusty-btrfs kernel: [ 0.135359] Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, full-width counters, Intel PMU driver. Jun 3 00:21:10 trusty-btrfs kernel: [ 0.135366] perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode ... Jun 3 08:47:13 trusty-btrfs kernel: [30388.364946] perf interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 ... Jun 4 12:10:17 trusty-btrfs kernel: [126403.543120] perf interrupt took too long (5019 > 5000), lowering kernel.perf_event_max_sample_rate to 25000 This doesn't happen all that often (at least, it didn't with 3.15-rc7 and below) so it would be useful if someone has suggestions for other things to try the next time I see it.
Additional to my comment above, this 'stuck at 800MHz' state persisted across multiple reboots into 3.15-rc8, 3.15-rc7 and 3.14.4 (five in total) using intel_pstate. I finally rebooted into the default 3.13.0-29-generic kernel and now have some frequency scaling (note that it's using the acpi-cpufreq driver), but the cores won't go beyond 2000 Mhz, ie they won't kick into turbo. I currently have speedstep enabled in the BIOS.
And additional to both my comments above, I rebooted from 3.13 into 3.15-rc8 and intel_pstate was still stuck at 800MHz. Then I did a powerdown and booted into 3.15-rc8 (without removing the laptop battery) and now things are back to normal. I'm still getting the "perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode" message in the syslog.
(In reply to Zhang Rui from comment #19) > (In reply to Doug Smythies from comment #18) > > There are comments in this bug report about CPU frequencies locking up at > > some rate, but it isn't clear how this happens. Somewhere and sometime ago, > > I mentioned one method. I'll mention it again: > > > > In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the > > intel_pstate driver: > > > > echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo > > > > All CPU's will now be locked at whatever frequency they were at when the > > above command was issued. > > I think you mean all cpu' will NOT be locked...? > No, I meant to say "locked at whatever frequency", and the issue persists with Kernel 3.15RC7 plus the 4 patches Dirk sent out a few days ago. Perhaps I should have said "locked at whatever pstate they were in when the command was issued". Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about 1.78 GHz; Executed the above mentioned turbo off command; Terminated the process that had CPU 7 slightly loaded; Started 12 CPU burning threads; Then: doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq 1800007 1800007 1800007 1800007 1800007 1800007 1800007 1800007 doug@s15:~/temp$ uptime 08:04:11 up 25 min, 3 users, load average: 12.06, 10.87, 6.57
(In reply to Doug Smythies from comment #23) > (In reply to Zhang Rui from comment #19) > > (In reply to Doug Smythies from comment #18) > > > There are comments in this bug report about CPU frequencies locking up at > > > some rate, but it isn't clear how this happens. Somewhere and sometime > ago, > > > I mentioned one method. I'll mention it again: > > > > > > In BIOS disable turbo mode, and boot as normal. Now, disable turbo for > the > > > intel_pstate driver: > > > > > > echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo > > > > > > All CPU's will now be locked at whatever frequency they were at when the > > > above command was issued. > > > > I think you mean all cpu' will NOT be locked...? > > > > No, I meant to say "locked at whatever frequency", and the issue persists > with Kernel 3.15RC7 plus the 4 patches Dirk sent out a few days ago. > Perhaps I should have said "locked at whatever pstate they were in when the > command was issued". > > Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about > 1.78 GHz; Executed the above mentioned turbo off command; Terminated the > process that had CPU 7 slightly loaded; Started 12 CPU burning threads; Then: > > doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq > 1800007 > 1800007 > 1800007 > 1800007 > 1800007 > 1800007 > 1800007 > 1800007 > doug@s15:~/temp$ uptime > 08:04:11 up 25 min, 3 users, load average: 12.06, 10.87, 6.57 This does not happen when turbo is enabled in the BIOS correct? When you have the CPU in the locked up condition and the working can you run the command rdmsr -x -a 0x199 Turning off turbo in the BIOS is probably not the right answer. I don't know why the pstate change requests are ignored in this case.
(In reply to Dirk Brandewie from comment #24) > (In reply to Doug Smythies from comment #23) ... > > Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about > > 1.78 GHz; Executed the above mentioned turbo off command; Terminated the > > process that had CPU 7 slightly loaded; Started 12 CPU burning threads; > Then: > > > > doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq > > 1800007 > > 1800007 > > 1800007 > > 1800007 > > 1800007 > > 1800007 > > 1800007 > > 1800007 > > doug@s15:~/temp$ uptime > > 08:04:11 up 25 min, 3 users, load average: 12.06, 10.87, 6.57 > This does not happen when turbo is enabled in the BIOS correct? That is correct, yes. > > When you have the CPU in the locked up condition and the working can you run > the command > rdmsr -x -a 0x199 I get nothing: doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq 2000023 2000023 2000023 2000023 2000023 2000023 2000023 2000023 doug@s15:~/temp$ sudo rdmsr -x -a 0x199 doug@s15:~/temp$ uptime 11:03:07 up 6 min, 3 users, load average: 7.81, 3.10, 1.17 > > Turning off turbo in the BIOS is probably not the right answer. Why not? I consider it to be exactly one right way to do it. Regardless, does it actually matter for the discussion? It is something that doesn't work as expected. > I don't know why the pstate change requests are ignored in this case. I didn't dig into it much, but here is what I did get: In this case, the intel_pstate driver never calculates that the target pstate need to increase). Example from "perf record" (every line is similar): consume 3348 [004] 1933.934957: power:pstate_sample: core_busy=58 scaled=95 state=21 mperf=159893 aperf=94055 freq=2000023 The highest frequency for pstate 21 should be higher, shouldn't it? For the above numbers, and with the default p gain of 20%, the target pstate will never change. With turbo turned on in the BIOS, "perf record" (starting from no load, so as to capture the transition through where it previously locked up), notice how in state 21, the actual CPU frequency goes higher previously (shown in the next line after the state got set to 21): test1 3234 [005] 456.836620: power:pstate_sample: core_busy=55 scaled=90 state=21 mperf=159820 aperf=89279 freq=1899351 test1 3234 [005] 456.848612: power:pstate_sample: core_busy=61 scaled=91 state=23 mperf=159788 aperf=98671 freq=2099500 Previously, I forgot to load the msr module first. What was actually requested, again: doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq 1700000 1700000 1700000 1700000 1700000 1700000 1700000 1700000 doug@s15:~/temp$ uptime 15:09:57 up 17 min, 3 users, load average: 9.99, 7.66, 3.78 doug@s15:~/temp$ sudo rdmsr -x -a 0x199 1000 1000 1000 1000 1000 1000 1000 1100 And for that one, "perf record": test1 3247 [005] 1127.403413: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159917 aperf=79959 freq=1700000 test1 3247 [005] 1127.415404: power:pstate_sample: core_busy=50 scaled=89 state=19 mperf=159895 aperf=79947 freq=1700000 test1 3247 [005] 1127.427395: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159883 aperf=79942 freq=1700000 test1 3247 [005] 1127.439386: power:pstate_sample: core_busy=50 scaled=89 state=19 mperf=159887 aperf=79943 freq=1700000 test1 3247 [005] 1127.451380: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159912 aperf=79956 freq=1700000 ... forever... This time the driver calculates a new target pstate back, but the frequency never changes.
I have reproduced the lockup Doug is describing on my Acer test system. Hopefully other vendors BIOS get it right when disabling turbo in the BIOS. Here is the patch that fixes the issue on my test system based on v3.15-rc8. commit ecba7c4439f55259911a8f9ee2f57a2636ac120f Author: Dirk Brandewie <dirk.j.brandewie@intel.com> Date: Tue May 20 14:57:21 2014 -0700 intel_pstate: don't touch turbo bit if trubo disabled in BIOS. If turbo is disabled in the BIOS bit 38 should be set in MSR_IA32_MISC_ENABLE register per section 14.3.2.1 of the SDM Vol 3 document 325384-050US Feb 2014. If this bit is set do *not* attempt to disable trubo via the MSR_IA32_PERF_CTL register. On some systems trying to disable turbo via MSR_IA32_PERF_CTL will cause subsequent writes to MSR_IA32_PERF_CTL not take affect, in fact reading MSR_IA32_PERF_CTL will not show the IDA/Turbo DISENGAGE bit(32) as set. A write of bit 32 to zero returns to normal operation. Signed-off-by: Dirk Brandewie <dirk.j.brandewie@intel.com> --- drivers/cpufreq/intel_pstate.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index eab8ccf..faf5b3a 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -97,6 +97,7 @@ struct cpudata { struct pstate_data pstate; struct vid_data vid; struct _pid pid; + int turbo_disabled; u64 prev_aperf; u64 prev_mperf; @@ -385,7 +386,7 @@ static void byt_set_pstate(struct cpudata *cpudata, int pstate) u32 vid; val = pstate << 8; - if (limits.no_turbo) + if (limits.no_turbo && !cpudata->turbo_disabled) val |= (u64)1 << 32; vid_fp = cpudata->vid.min + mul_fp( @@ -452,7 +453,7 @@ static void core_set_pstate(struct cpudata *cpudata, int pstate) u64 val; val = pstate << 8; - if (limits.no_turbo) + if (limits.no_turbo && !cpudata->turbo_disabled) val |= (u64)1 << 32; wrmsrl_on_cpu(cpudata->cpu, MSR_IA32_PERF_CTL, val); @@ -786,6 +787,7 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy) { struct cpudata *cpu; int rc; + u64 misc_en; rc = intel_pstate_init_cpu(policy->cpu); if (rc) @@ -793,6 +795,10 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy) cpu = all_cpu_data[policy->cpu]; + rdmsrl(MSR_IA32_MISC_ENABLE, misc_en); + if (misc_en & MSR_IA32_MISC_ENABLE_TURBO_DISABLE) + cpu->turbo_disabled = 1; + if (!limits.no_turbo && limits.min_perf_pct == 100 && limits.max_perf_pct == 100) policy->policy = CPUFREQ_POLICY_PERFORMANCE;
updated patch to cover case where the processor is fused to have max and turbo P states as the same and warn if the user tries to enable turbo via sysfs and it is disabled commit c35b542da1845cab2d26e1ffc9123457fb256120 Author: Dirk Brandewie <dirk.j.brandewie@intel.com> Date: Tue May 20 14:57:21 2014 -0700 intel_pstate: don't touch turbo bit if turbo disabled or unavailable. If turbo is disabled in the BIOS bit 38 should be set in MSR_IA32_MISC_ENABLE register per section 14.3.2.1 of the SDM Vol 3 document 325384-050US Feb 2014. If this bit is set do *not* attempt to disable trubo via the MSR_IA32_PERF_CTL register. On some systems trying to disable turbo via MSR_IA32_PERF_CTL will cause subsequent writes to MSR_IA32_PERF_CTL not take affect, in fact reading MSR_IA32_PERF_CTL will not show the IDA/Turbo DISENGAGE bit(32) as set. A write of bit 32 to zero returns to normal operation. Also deal with the case where the processor does not support turbo and the BIOS does not report the fact in MSR_IA32_MISC_ENABLE but does report the max and turbo P states as the same value. Signed-off-by: Dirk Brandewie <dirk.j.brandewie@intel.com> --- drivers/cpufreq/intel_pstate.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index c034cde..ab554c4 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -132,6 +132,7 @@ static struct pstate_funcs pstate_funcs; struct perf_limits { int no_turbo; + int turbo_disabled; int max_perf_pct; int min_perf_pct; int32_t max_perf; @@ -294,7 +295,10 @@ static ssize_t store_no_turbo(struct kobject *a, struct attribute *b, if (ret != 1) return -EINVAL; limits.no_turbo = clamp_t(int, input, 0 , 1); - + if (limits.turbo_disabled) { + pr_warn("Turbo disabled by BIOS or unavailable on processor\n"); + limits.no_turbo = limits.turbo_disabled; + } return count; } @@ -388,7 +392,7 @@ static void byt_set_pstate(struct cpudata *cpudata, int pstate) u32 vid; val = pstate << 8; - if (limits.no_turbo) + if (limits.no_turbo && !limits.turbo_disabled) val |= (u64)1 << 32; vid_fp = cpudata->vid.min + mul_fp( @@ -455,7 +459,7 @@ static void core_set_pstate(struct cpudata *cpudata, int pstate) u64 val; val = pstate << 8; - if (limits.no_turbo) + if (limits.no_turbo && !limits.turbo_disabled) val |= (u64)1 << 32; wrmsrl_on_cpu(cpudata->cpu, MSR_IA32_PERF_CTL, val); @@ -756,7 +760,7 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy) limits.min_perf = int_tofp(1); limits.max_perf_pct = 100; limits.max_perf = int_tofp(1); - limits.no_turbo = 0; + limits.no_turbo = limits.turbo_disabled; return 0; } limits.min_perf_pct = (policy->min * 100) / policy->cpuinfo.max_freq; @@ -799,6 +803,7 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy) { struct cpudata *cpu; int rc; + u64 misc_en; rc = intel_pstate_init_cpu(policy->cpu); if (rc) @@ -806,8 +811,13 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy) cpu = all_cpu_data[policy->cpu]; - if (!limits.no_turbo && - limits.min_perf_pct == 100 && limits.max_perf_pct == 100) + rdmsrl(MSR_IA32_MISC_ENABLE, misc_en); + if (misc_en & MSR_IA32_MISC_ENABLE_TURBO_DISABLE || + cpu->pstate.max_pstate == cpu->pstate.turbo_pstate) { + limits.turbo_disabled = 1; + limits.no_turbo = 1; + } + if (limits.min_perf_pct == 100 && limits.max_perf_pct == 100) policy->policy = CPUFREQ_POLICY_PERFORMANCE; else policy->policy = CPUFREQ_POLICY_POWERSAVE;
My test computer motherboard (with i7-2600K CPU @ 3.40GHz) is Asus, and I last updated the BIOS on 2013.11.15. I tried, and confirm, Dirk's patch (version 2). Thanks and sorry for the slight diversion on this bug report.
Is my issue (cores sometimes get stuck at 800MHz using pstate even after reboot) related to this bug? The issue *might* be related to the power cord. The last time this happened, a complete shutdown and reboot didn't fix the issue, but removing and replugging the power cord and then a complete shutdown and reboot did fix it.
Rocko are you still having this issue? The patch above has been merged.
No, I haven't seen the cores getting stuck at low frequency in a long time now, IIRC not since I reported it in June (comment 29). For info, I've been using 3.16.0/1/2/3 since each one came out and have just switched to 3.17-rc6.
Unfourtunately, I just got stuck with 620mhz after resume-from-ram with linux-3.18.3, so I guess it still hasn't been fixed.
Ok, this doesn seem to be strictly linked with suspend-to-ram - I just experienced all cores stuck at 625mhz after 4h of uptime (cold boot) without any suspend before. The only interesting entries in syslog are: [11652.186969] perf interrupt took too long (2558 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 [14809.974491] dell_wmi: Received unknown WMI event (0x12)
Hi Clemens, Is it possible you are experiencing thermal issues? Can you please try to duplicate with a vanilla upstream kernel and send me the exact duplication steps? Thanks.
I am 100% sure those issue were not triggered by thermal issues - the system showed the behaviour after suspend from ram - where the CPU sitting withing idle without power for hours. Also, there were not the usual thermal messages which showed up when loading all 4 cores for hours (during FPGA synthesis). However, as the notebook dies a few weeks ago, I can't test it anymore :/
I have Dell Precision M4600 laptop and have a similar issue. The processor is 4-core i7 Sandy Bridge: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 42 model name : Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz stepping : 7 microcode : 0x29 cpu MHz : 803.906 cache size : 6144 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt bugs : bogomips : 4789.08 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: After the laptop is freshly booted from poweroff, full CPU frequency range from 800 MHz to 2.4+ GHz (including turbo boost) is used, depending on load. After I suspend to RAM and resume on AC power, everything is almost ok, almost full frequency range used, with the exception of turbo-boost. After I suspend to RAM on AC power or battery and *resume on battery*, all cores of the processor are stuck at 800 MHz. This behavior is repeatable - it happens always on resuming on battery and never happened to me on resuming on AC. Upgrade to recent BIOS A16 for this model did not help. Somewhere on the Internet I found a workaround to overwrite /sys/devices/system/cpu/intel_pstate/max_perf_pct file twice, once to change the value for something lower than 100, then to bring it back to 100 and it unlocked full frequency range. I observed the problem on only some kernel versions (I haven't tried all of them but): 3.16.x - buggy, but workaround works 3.18.12 - ok (I was so happy to see it fixed) 3.19.4 to 3.19.8 - all seem ok (currently I'm running 3.19.8) 4.0.1, 4.0.2 - buggy again (regression?), and the workaround seem to not work anymore, but I haven't tried hard I'm using Linux Mint 17.1 and taking kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/
For sure those issues are not thermal- nor CPU load-related for me. When it happens, lm-sensors reports temperatures well below 50 C for all cores and the fans are either off (total silence) or at the first speed step. Downscaling and locking frequency to 800 MHz never happens when I load the laptop heavily with multithreaded tasks - if I do it when the bug is not effective and full range is available, the CPU frequency rather goes up to 3.2 GHz for all cores (turbo-boost), fans go to full speed, but it never overheats. I clean fans/heatsinks from dust regularly ;) My current sensors output when I'm writing this: acpitz-virtual-0 Adapter: Virtual device temp1: +25.0°C (crit = +107.0°C) coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +43.0°C (high = +86.0°C, crit = +100.0°C) Core 0: +43.0°C (high = +86.0°C, crit = +100.0°C) Core 1: +39.0°C (high = +86.0°C, crit = +100.0°C) Core 2: +41.0°C (high = +86.0°C, crit = +100.0°C) Core 3: +37.0°C (high = +86.0°C, crit = +100.0°C) i8k-virtual-0 Adapter: Virtual device fan1: 2571 RPM fan2: 2609 RPM temp1: +42.0°C temp2: +39.0°C temp3: +38.0°C temp4: +38.0°C For me the problem mentioned here is suspend/resume related.
@Piotr: Could you please supply a little additional information: Both before and after suspend, the output from: cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor cat /sys/devices/system/cpu/intel_pstate/no_turbo cat /sys/devices/system/cpu/intel_pstate/max_perf_pct cat /sys/devices/system/cpu/intel_pstate/min_perf_pct Also the before and after output from: sudo turbostat -d sleep 10 Do you use the generic or low latency kernels from the PPA? (I assume 64 bit).
Thanks for fast reply. I use generic ones. I'll reboot to 4.0.2 and post the results.
I don't have turbostat - where to get linux-tools package for kernel 4.0.2? I managed to get a creenshot of i7z, though. Fresh boot on AC: pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor powersave powersave powersave powersave powersave powersave powersave powersave pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo 0 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct 100 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct 22 Cpu speed from cpuinfo 2394.00Mhz cpuinfo might be wrong if cpufreq is enabled. To guess correctly try estimating via tsc Linux's inbuilt cpu_khz code emulated now True Frequency (without accounting Turbo) 2394 MHz CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4] TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25]) Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 35x/34x/32x/32x Real Current Frequency 2045.84 MHz [99.75 x 20.51] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % C7 % Temp Core 1 [0]: 2045.84 (20.51x) 10.8 38.8 1 0 51 60 Core 2 [1]: 2038.05 (20.43x) 4.26 10.6 2.02 0 83.8 60 Core 3 [2]: 1399.97 (14.03x) 2.72 2.5 1 0 94.9 60 Core 4 [3]: 1513.23 (15.17x) 2.05 1.79 1 0 95.9 60 C0 = Processor running without halting C1 = Processor running with halts (States >C0 are power saver) C3 = Cores running with PLL turned off and core cache turned off C6 = Everything in C3 + core state saved to last level cache Above values in table are in percentage over the last 1 sec [core-id] refers to core-id number in /proc/cpuinfo 'Garbage Values' message printed when garbage values are read Ctrl+C to exit After unplugging: pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor powersave powersave powersave powersave powersave powersave powersave powersave pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo 0 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct 100 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct 22 Cpu speed from cpuinfo 2394.00Mhz cpuinfo might be wrong if cpufreq is enabled. To guess correctly t Linux's inbuilt cpu_khz code emulated now True Frequency (without accounting Turbo) 2394 MHz CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz Socket [0] - [physical cores=4, logical cores=8, max online cores TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25 Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 35x/34x Real Current Frequency 1411.38 MHz [99.75 x 14.15] (Max of below Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% Core 1 [0]: 1411.38 (14.15x) 9.45 30.2 Core 2 [1]: 1175.90 (11.79x) 6.05 5.15 Core 3 [2]: 1132.88 (11.36x) 6.97 6.57 Core 4 [3]: 1203.24 (12.06x) 3.72 3.97 C0 = Processor running without halting C1 = Processor running with halts (States >C0 are power saver) C3 = Cores running with PLL turned off and core cache turned off C6 = Everything in C3 + core state saved to last level cache Above values in table are in percentage over the last 1 sec [core-id] refers to core-id number in /proc/cpuinfo 'Garbage Values' message printed when garbage values are read Ctrl+C to exit AFTER suspend/resume. I must take back what I said before - this doesn't seem to happen always. I had to try twice to trigger the problem. pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor powersave powersave powersave powersave powersave powersave powersave powersave pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo 0 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct 100 pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct 22 Linux's inbuilt cpu_khz code emulated now True Frequency (without accounting Turbo) 2394 MHz CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4] TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25]) Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 35x/34x/32x/32x Real Current Frequency 798.04 MHz [99.75 x 8.00] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % C7 % Temp Core 1 [0]: 797.83 (8.00x) 3.71 3.18 1 0 94.6 59 Core 2 [1]: 797.97 (8.00x) 3.44 4.35 1 0 93.5 58 Core 3 [2]: 797.86 (8.00x) 6.43 10.7 0 0 87.2 58 Core 4 [3]: 798.04 (8.00x) 1.15 1.36 0 0 98.3 58 The multiplier is stuck at 8.00x and never changes.
(In reply to Piotr Kołaczkowski from comment #36) > I observed the problem on only some kernel versions (I haven't tried all of > them but): > > 3.16.x - buggy, but workaround works > 3.18.12 - ok (I was so happy to see it fixed) > 3.19.4 to 3.19.8 - all seem ok (currently I'm running 3.19.8) > 4.0.1, 4.0.2 - buggy again (regression?), and the workaround seem to not > work anymore, but I haven't tried hard That is useful thanks. Resume from suspend is not working at all for me with kernels 4.1RC1 or RC2 or RC3. Upon resume, only CPU 0 comes back on-line. It works fine for me with Kernel 4.0. after resume from suspend, turbo works as expected as does the normal frequency verses load curve. Shortly, I will look at the post added while I was doing this one, however I do not like i7z and do not trust what is ways.
If you give me some hints how to get linux-tools installed on my system, I can try with turbostat. Is there some special ppa where I an install it from, or can I get the sources somewhere and compile? i7z frequency looks correct - when it is stuck at 800 MHz, the laptop is visibly much slower. Particularly CPU heavy tasks like compilation take "forever". "It works fine for me with Kernel 4.0. after resume from suspend, turbo works as expected as does the normal frequency verses load curve" For me the critical thing triggering this is resuming on battery. If I resume on AC power, everything is ok.
I recall having problems similar to that with some much earlier kernels < 3.10, which didn't use intel_pstate. But that was long time ago when I was using Ubuntu and I can't reproduce that setup anymore. Maybe this particular hardware or BIOS in this laptop has some weird bug in it that shows up with only some kernels?
(In reply to Piotr Kołaczkowski from comment #42) > If you give me some hints how to get linux-tools installed on my system, I > can try with turbostat. Is there some special ppa where I an install it > from, or can I get the sources somewhere and compile? turbostat source is included in the kernel source (tools/power/x86/turbostat/turbostat.c), you just have to go to its directory and build it. I did e-mail you a link to where I have a recently compiled turbostat on my web site. Currently, I am bisecting the kernel to determine where suspend broke for my system. However, for every "bad" I forget to re-boot, and then only have one working CPU for the next compile.
This is a bit of a tangent, but: (In reply to Doug Smythies from comment #41) > Resume from suspend is not working at all for me with kernels 4.1RC1 or RC2 > or RC3. Upon resume, only CPU 0 comes back on-line. For my system, suspend was broken by: 3c18d447b3b36a8d3c90dc37dfbd363cdb685d0a "sched/core: Check for available DL bandwidth in cpuset_cpu_inactive()" And is fixed by: 533445c6e533 "sched/core: Fix regression in cpuset_cpu_inactive() for suspend" which I think, but am not sure, is queued for Kernel 4.1RC4.
(In reply to Piotr Kołaczkowski from comment #40) > AFTER suspend/resume. > I must take back what I said before - this doesn't seem to happen always. I > had to try twice to trigger the problem. Suggesting some sort of race condition or other timing sensitive event, which might explain why, or so it seems, some processors do not experience this issue, but some do.
Today I installed kernel 4.0.3 and compiled turbostat for it. So far, after more than 10 suspends/resumes, both on AC or battery, everything works fine including turbo-boost (frequencies go above 2.4 GHz under load, so it must be kicking in). However, now I have disabled Optimus in BIOS and I'm running on Intel Video driver. Previous problems happened when I had Optimus on nvidia-prime was in mode "nvidia", not "intel". I'll reboot to Optimus and do the same experiment with multiple suspend/resume.
Just for the record: my lattitide E6320 didn't have Optimus, only the integrated intel hd graphics.
Ah, I messed up my previous post a little bit. I had Optimus *enabled* and I was running on Intel Video driver, i.e. prime-select intel. It is not possible to run this laptop on Intel card, without turning on optimus.
I found something very interesting. I almost thought everything was fine, but after several suspend/resumes on kernel 4.0.3 I noticed now my CPU is capped at multiplier 18x, that is 1800 MHz. Much better than the previous 800 MHz, but still far from perfect. turbostate before suspend (but CPU was not loaded): turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org> CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7) CPUID(6): APERF, DTS, PTM, EPB RAPL: 1456 sec. Joule Counter Range, at 45 Watts cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800 8 * 100 = 800 MHz max efficiency 24 * 100 = 2400 MHz TSC frequency cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled) cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7) cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223 32 * 100 = 3200 MHz max turbo 4 active cores 32 * 100 = 3200 MHz max turbo 3 active cores 34 * 100 = 3400 MHz max turbo 2 active cores 35 * 100 = 3500 MHz max turbo 1 active cores cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced) cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.) cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.) cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked) cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled) cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled) cpu0: MSR_PP0_POLICY: 0 cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_PP1_POLICY: 0 cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C) cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x882d0000 (55 C) cpu0: MSR_IA32_THERM_STATUS: 0x882d0000 (55 C +/- 1) cpu1: MSR_IA32_THERM_STATUS: 0x882f0000 (53 C +/- 1) cpu2: MSR_IA32_THERM_STATUS: 0x882f0000 (53 C +/- 1) cpu3: MSR_IA32_THERM_STATUS: 0x88340000 (48 C +/- 1) Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 8 0.97 819 2395 0 1.74 0.02 0.00 97.27 55 55 5.01 0.04 5.46 80.17 4.01 0.33 0.09 0 0 16 1.98 820 2395 1 1.02 0.06 0.00 96.94 55 55 5.01 0.04 5.46 80.17 4.01 0.33 0.09 0 4 0 0.05 808 2395 1 2.94 1 1 20 2.43 826 2395 1 0.69 0.00 0.00 96.88 54 1 5 1 0.07 835 2395 1 3.05 2 2 11 1.35 820 2395 1 0.56 0.01 0.00 98.08 54 2 6 0 0.02 806 2395 1 1.89 3 3 10 1.24 810 2395 1 1.58 0.02 0.00 97.16 54 3 7 5 0.60 802 2395 1 2.22 10.003878 sec After resume on battery: turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org> CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7) CPUID(6): APERF, DTS, PTM, EPB RAPL: 1456 sec. Joule Counter Range, at 45 Watts cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800 8 * 100 = 800 MHz max efficiency 24 * 100 = 2400 MHz TSC frequency cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled) cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7) cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223 32 * 100 = 3200 MHz max turbo 4 active cores 32 * 100 = 3200 MHz max turbo 3 active cores 34 * 100 = 3400 MHz max turbo 2 active cores 35 * 100 = 3500 MHz max turbo 1 active cores cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance) cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.) cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.) cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked) cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled) cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled) cpu0: MSR_PP0_POLICY: 0 cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_PP1_POLICY: 0 cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C) cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x881f0000 (69 C) cpu0: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1) cpu1: MSR_IA32_THERM_STATUS: 0x88200000 (68 C +/- 1) cpu2: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1) cpu3: MSR_IA32_THERM_STATUS: 0x88240000 (64 C +/- 1) Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 1138 63.41 1795 2395 0 20.20 1.71 0.14 14.54 70 70 0.19 0.15 0.10 0.02 17.13 12.87 0.09 0 0 1457 81.20 1794 2395 0 12.96 1.46 0.18 4.20 70 70 0.19 0.15 0.10 0.02 17.13 12.87 0.09 0 4 959 53.40 1795 2395 0 40.76 1 1 1170 65.17 1796 2395 0 18.63 1.65 0.25 14.30 67 1 5 1078 60.01 1796 2395 0 23.79 2 2 1155 64.33 1795 2395 0 14.90 1.96 0.07 18.74 67 2 6 1186 66.03 1795 2395 0 13.20 3 3 1115 62.11 1795 2395 0 15.16 1.76 0.04 20.94 60 3 7 988 55.05 1795 2395 0 22.22 10.002551 sec
And state of sensors (machine under load, compiling some project in background): acpitz-virtual-0 Adapter: Virtual device temp1: +25.0°C (crit = +107.0°C) coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +65.0°C (high = +86.0°C, crit = +100.0°C) Core 0: +65.0°C (high = +86.0°C, crit = +100.0°C) Core 1: +62.0°C (high = +86.0°C, crit = +100.0°C) Core 2: +61.0°C (high = +86.0°C, crit = +100.0°C) Core 3: +56.0°C (high = +86.0°C, crit = +100.0°C) i8k-virtual-0 Adapter: Virtual device Processor Fan: 0 RPM Video Fan: 0 RPM CPU: +64.0°C Ambient: +43.0°C SODIMM: +44.0°C SODIMM: +44.0°C
Putting the laptop back on AC, without reboot, *does not unlock it*.
Putting the laptop back on AC and suspending / resuming unlocks it partially. The maximum frequency is 2.4 GHz, so no turbo. turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org> CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7) CPUID(6): APERF, DTS, PTM, EPB RAPL: 1456 sec. Joule Counter Range, at 45 Watts cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800 8 * 100 = 800 MHz max efficiency 24 * 100 = 2400 MHz TSC frequency cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled) cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7) cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223 32 * 100 = 3200 MHz max turbo 4 active cores 32 * 100 = 3200 MHz max turbo 3 active cores 34 * 100 = 3400 MHz max turbo 2 active cores 35 * 100 = 3500 MHz max turbo 1 active cores cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance) cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.) cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.) cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked) cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled) cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled) cpu0: MSR_PP0_POLICY: 0 cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_PP1_POLICY: 0 cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C) cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x881f0000 (69 C) cpu0: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1) cpu1: MSR_IA32_THERM_STATUS: 0x88240000 (64 C +/- 1) cpu2: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1) cpu3: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1) Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 622 25.95 2399 2399 0 24.27 2.16 0.00 47.63 69 69 0.00 0.00 0.00 0.00 16.17 12.78 0.09 0 0 681 28.35 2402 2402 1 21.96 0.50 0.00 49.19 65 69 0.00 0.00 0.00 0.00 16.17 12.78 0.09 0 4 600 24.98 2400 2400 1 25.29 1 1 593 24.72 2400 2400 1 48.25 2.06 0.00 24.97 69 1 5 1234 51.40 2400 2400 1 21.57 2 2 687 28.66 2398 2398 1 16.72 3.09 0.00 51.53 65 2 6 392 16.35 2397 2397 1 29.01 3 3 551 22.99 2397 2397 1 9.24 2.99 0.00 64.78 58 3 7 242 10.09 2395 2395 1 22.06 10.002048 sec
Resuming again on battery brought it back to 1.8 GHz limit. I repeated that 6 times and the maximum achievable frequency was always 1.8 GHz. Then I resumed again on AC 6 times, and all the times the limit was 2.4 GHz. The good news is that so far I *never* got 800 MHz limit with 4.0.3 kernel, despite suspending/resuming more than 25 times since last reboot. On 4.0.2 that condition was pretty easy to trigger. Also, I'm pretty impressed with the overall stability of the suspend/resume process compared to 3.18.x and 3.19.x series. With those earlier kernels there was some weird intermittent bug preventing my laptop going to sleep sometimes. It looked as if it went to sleep fine (including turning off display, downspinning hdd, turning off fans, turning off wireless / bt) but immediately resumed afterwards (and funny - it was fully operational then, so I could just hit "suspend" once again, and often it did that fine). I also had a permanent freeze once after resuming on 3.19.x. So far on 4.0.3 none of those problems ever happened, so I'm now staying on that kernel and will observe it.
turbostat after a fresh reboot, this time under load. This is how it should look like :) turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org> CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7) CPUID(6): APERF, DTS, PTM, EPB RAPL: 1456 sec. Joule Counter Range, at 45 Watts cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800 8 * 100 = 800 MHz max efficiency 24 * 100 = 2400 MHz TSC frequency cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled) cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7) cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223 32 * 100 = 3200 MHz max turbo 4 active cores 32 * 100 = 3200 MHz max turbo 3 active cores 34 * 100 = 3400 MHz max turbo 2 active cores 35 * 100 = 3500 MHz max turbo 1 active cores cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced) cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.) cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.) cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked) cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled) cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled) cpu0: MSR_PP0_POLICY: 0 cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_PP1_POLICY: 0 cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C) cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88110000 (83 C) cpu0: MSR_IA32_THERM_STATUS: 0x88150000 (79 C +/- 1) cpu1: MSR_IA32_THERM_STATUS: 0x88110000 (83 C +/- 1) cpu2: MSR_IA32_THERM_STATUS: 0x88120000 (82 C +/- 1) cpu3: MSR_IA32_THERM_STATUS: 0x88170000 (77 C +/- 1) Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 2159 68.07 3172 2398 0 24.47 0.48 0.03 6.95 86 86 0.04 0.06 0.00 0.00 50.02 45.72 0.09 0 0 2636 82.94 3178 2400 1 12.28 0.93 0.12 3.73 82 86 0.04 0.06 0.00 0.00 50.02 45.72 0.09 0 4 1800 56.66 3177 2400 1 38.56 1 1 2558 80.50 3178 2400 1 17.60 0.03 0.00 1.88 86 1 5 2446 77.01 3176 2398 1 21.09 2 2 1960 61.82 3170 2398 1 26.40 0.66 0.00 11.12 78 2 6 2266 71.57 3166 2397 1 16.64 3 3 2140 67.53 3169 2397 1 21.08 0.32 0.00 11.06 83 3 7 1471 46.51 3162 2395 1 42.09
Now, all on AC, I suspended/resumed multiple times (over 5x) and turbo-boost always worked (checked under load - frequencies about 3.2 GHz reported on all cores). This is really weird. Now I suspended/resumed multiple times on battery, and turboboost works fine. So I still have no idea what triggers this. However, when the laptop goes into that "wrong" state, it becomes very repeatable: maximums are 2.4 GHz on AC and 1.8 GHz on battery and it remembers its state until the next reboot. Any ideas how to debug it further? Is it possible to enable more logging?
I did another reboot on AC, tried doing multiple suspends both on AC and battery, and had no luck reproducing it (still kernel 4.0.3). Then I booted the laptop on battery power. Turboboost was working fresh after reboot. Then I suspended it twice on battery, and after the second resume I have my laptop again in "invalid state" with limit 1.8 GHz. :/ Both min_perf_pct and max_perf_pct are 100. no_turbo = 0. I tried the old workaround trick with overwriting the contents of max_perf_pct. I issued the following: m4600 intel_pstate # echo 99 >> max_perf_pct // waited a few seconds and then m4600 intel_pstate # echo 100 >> max_perf_pct Surprising, now my max frequency is... 2.6 GHz, multiplier 26x. No suspend/resume was needed to unlock it, AC was never connected in this session since reboot, still working on battery.
Connected to AC, suspended / resumed. Max freq was at 2.4 GHz (so lower than previous 2.6 GHz on battery). Then performed the same "trick" with max_perf_pct and it unlocked the freq range to at least 3.2 GHz.
> Both min_perf_pct and max_perf_pct are 100. Huh? Are you somehow using the performance mode governor now? Do you know how min_perf_pct got to 100? These are potentially significant: > cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced) > cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance) > cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled) > cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled) Are you comfortable compiling the kernel? It might be time to acquire some trace data, but a patch (attachment 169891 [details]) is needed.
On my system (no tlp, no thermald, no battery) the MSR_IA32_ENERGY_PERF_BIAS setting is not restored upon resume from suspend. It is "balanced" before the suspend and "performance" after. Kernel 4.1rc3 + 533445c6e533.
I have no tlp or any other extra pm packages installed. I booted to older 4.0.2 kernel (on AC) and observed that: Initially after logging in I got: m4600 intel_pstate # cat min_perf_pct 100 m4600 intel_pstate # cat max_perf_pct 100 But some time later (a minute?), when double-checking I got: m4600 intel_pstate # cat max_perf_pct 100 m4600 intel_pstate # cat min_perf_pct 22 I didn't do *anything* to manually change those settings.
> Are you comfortable compiling the kernel? It might be time to acquire some trace data, but a patch (attachment 169891 [details]) is needed. I did that last time about 10 years ago. Sounds fun. No problem. :) So please confirm this plan is good: 1. download sources from kernel.org 2a. apply three Ubuntu patches from ppa/mainline, for 4.0.3 kernel version 2b. apply any other patches from you 3. make oldconfig 4. make 5. somehow install new kernel to mint - I have no idea how one does that - any hints? make install + update-grub? Or something else?
Fresh reboot, immediately after logging into the graphical session, I issued the following commands: pkolaczk@m4600 ~ $ sudo su [sudo] password for pkolaczk: m4600 pkolaczk # cd /sys/devices/system/cpu/intel_pstate/ m4600 intel_pstate # while true; do cat min_perf_pct; sleep 1; done 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 22 22 22 22 22 22 22 22 Leaving it running and then suspending/resuming on AC twice: 22 22 22 22 22 22 100 100 100 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 100 100 100 22 22 22 22 Is it expected, or is something in Ubuntu/Mint changing governors? What to look for?
I also checked I *don't* have thermald installed.
(In reply to Piotr Kołaczkowski from comment #63) > Fresh reboot, immediately after logging into the graphical session, I issued > the following commands: > > Is it expected, or is something in Ubuntu/Mint changing governors? > What to look for? Yes, There is a script that is run on startup that sleeps for 1 minute, then sets "ondemand" mode if it exits, or "powersave" mode if it exists. see /etc/init.d/ondemand (at least I think for your distro). How it gets restored after suspend I do not know.
Oh yes, you're right. There is a script that does exactly that. I'll disable it temporarily and retry the experiments. In the meantime, I'm compiling a patched kernel 4.0.3 with more tracing.
(In reply to Piotr Kołaczkowski from comment #66) > Oh yes, you're right. There is a script that does exactly that. I'll disable > it temporarily and retry the experiments. In the meantime, I'm compiling a > patched kernel 4.0.3 with more tracing. You should leave the script enabled, as you want "powersave" mode (or set it manually). Once you have the patched kernel, create your failure scenario and run: sudo ~/bin/perf record -a --event=power:pstate_sample sleep 300 in the 5 minute sample time do whatever loading of the CPU that shows the issue and do the thing that unlocks it or whatever (adjust the 300 seconds to longer if needed). Either post the resulting pref.data here or e-mail it to me directly, and I'll post process it.
Sounds good. In the meantime, which parts of the kernel sources should I look into to get better understanding of intel_pstate driver? Can you point me to some papers / documentation on this? I'm eager to try out any patches when you have them, but I'd like to also get some better understanding how it all works :)
(In reply to Piotr Kołaczkowski from comment #68) > Sounds good. In the meantime, which parts of the kernel sources should I > look into to get better understanding of intel_pstate driver? drivers/cpufreq/intel_pstate.c > Can you point > me to some papers / documentation on this? Documentation/cpu-freq/intel-pstate.txt
When building kernel with fakeroot make-kpkg --initrd kernel_image kernel_headers modules_image I ran into this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745686 restore_upstream_debianization test ! -f scripts/package/builddeb.kpkg-dist || mv -f scripts/package/builddeb.kpkg-dist scripts/package/builddeb test ! -f scripts/package/Makefile.kpkg-dist || mv -f scripts/package/Makefile.kpkg-dist scripts/package/Makefile /usr/bin/make INSTALL_MOD_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3 \ INSTALL_FW_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3/lib/firmware/4.0.3 \ INSTALL_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot install make[2]: Wejście do katalogu `/home/pkolaczk/Pobrane/linux-4.0.3' scripts/kconfig/conf --silentoldconfig Kconfig make[2]: Opuszczenie katalogu `/home/pkolaczk/Pobrane/linux-4.0.3' make[2]: Wejście do katalogu `/home/pkolaczk/Pobrane/linux-4.0.3' sh ./arch/x86/boot/install.sh 4.0.3 arch/x86/boot/bzImage \ System.map "/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot" run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.0.3 /home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot/vmlinuz-4.0.3 /etc/kernel/postinst.d/apt-auto-removal: 84: /etc/kernel/postinst.d/apt-auto-removal: cannot create /etc/apt/apt.conf.d//01autoremove-kernels.dpkg-new: Permission denied Any quick workaround?
A lot of the published methods for compiling the kernel have issues. Here is how I compile the kernel (from the linux directory of the source): 1.) steal the ubuntu kernel configuration: cp /boot/config-4.1.0-040100rc3-generic .config 2.) disable excessive debug settings in ubuntu config: scripts/config --disable DEBUG_INFO 3.) Compile the kernel, with a localized name: (I always time the compile, and I have 8 CPUs.) time make -j8 deb-pkg LOCALVERSION=-fix 4.) Install: sudo dpkg -i ../linux-headers-4.1.0-rc3-fix_4.1.0-rc3-fix-33_amd64.deb sudo dpkg -i ../linux-image-4.1.0-rc3-fix_4.1.0-rc3-fix-33_amd64.deb 5.) Re-boot and select the desired kernel from the grub menu (I always have grub set to display its menu for about 20 seconds).
Thanks, that did it! I'm writing this from my new kernel. Will ping you soon when I have some good perf.data.
(In reply to Doug Smythies from comment #60) > On my system (no tlp, no thermald, no battery) the MSR_IA32_ENERGY_PERF_BIAS > setting is not restored upon resume from suspend. It is "balanced" before > the suspend and "performance" after. Kernel 4.1rc3 + 533445c6e533. Addendum 1: Only for CPU 0. The other CPUs are fine. Addendum 2: Same as of Kernel 4.1RC4.
Kernel 4.1-rc4 did not help. Today I booted laptop on battery once again on kernel 4.1.0-rc4, then suspended and resumed and found it locked at 800 MHz. Interestingly I observed that immediately after resume, max frequency was 1.8 GHz and it changed to 800 MHz a few seconds later, so I guess it was triggered by switching the governor to powersave by the script.
Everybody: Piotr sent me some trace data via e-mail. Executive Summary: . There are multiple problems. . There are problems with math lockup within the intel_pstate driver. . There are problems with processor configuration upon exit from suspend. . There might be problems (not sure on this one) where the intel_pstate driver thinks the new target pstate is the same as previously sent, so it doesn't send the new one. However in the meantime there has been a suspend and the target pstate is not what the driver thinks it is. . Apparently the processor can get into a state where the pstate isn't supposed to even be possible. Suggestions: . Math lockup: Use my proposed patch set instead. The math can not lock up. . Processor Configuration: We should synchronize ourselves with kernel 4.1RC4. There was a bunch of activity as recently as Monday morning on this, with RC4 delayed to later on Monday as a result. However, I am aware of one mistake, that I guess will carry over to RC5. . Confused intel_pstate driver: Somehow it needs to re-initialize properly, which may or may not occur as a result of fixing the above. Details: (with Piotr's e-mail copied herein): > I managed to reproduce the problem. > This time it locked at 800 MHz for the first time (!) > I suspended it. This is a math problem within the current version of the intel_pstate driver. For your particular processor, and for the default settings used, it will never break out of a target pstate of 8 under the given conditions of your trace. > perf1.data was recorded in the following session: > 1. fresh boot on battery > 2. started perf recording > 3. started compilation of some big project to load CPU; > got freq reported by i7z up to 3.2 GHz > 4. suspended > 5. resumed, frequency locked at 800 MHz The trace data reveals: . Only CPU 0 came back on-line. This is the same problem I had, and is fixed in kernel 4.1RC4 . MPERF, APERF, and TSC are ridiculous after the suspend. Obvioulsy, the 64 bit counters reset during the suspend. This situation may correct itself if the intel_pstate driver is re-initialized properly. . CPU 0 seems to be in performance mode for a few seconds afterwards. However the CPU frequency suggests the target pstate is 18. I suspect the intel_pstate driver never sends the new target pstate because it thinks it is unchanged (35 was the last one sent before the suspend). . The frequency is not locked at 800 MHz. After a number of seconds it ends up in the 600 to 700 Mhz range. (???) . I assume that at this point the governor was set back to powersave. . at some point performance mode is set again, and the CPU frequency does respond, but it doesn't make much sense. Eventually it does settle to 2.6 GHz. . Then powersave again. And 600 to 700 Mhz again. There is a perturbation that causes a 1 sample increase in frequency to 853 MHz. > 6. tried doing the echo 99 >> max_perf_pct trick a few times, > but it didn't help this time No, for this particular lockup scenario, it wouldn't. > 7. recording finished Time = 325.421 seconds. > perf2.data: > 8. started recording again (no reboot since the previous session, > so still stuck at 800 MHz) Time = 393 seconds. All CPUs are in the trace. What happened in between? No, frequency stuck at 600 to 700 MHz. > 9. put laptop on AC It seems to go into performance mode at some point. The CPU frequency goes up, but jumps around, however 2.6 GHz is typical. > 10. suspended > 11. resumed, frequencies went back to 3.2. GHz First it (CPU 0) goes to 2.4GHz. Again, I think the driver never sends the target pstate because it thinks it didn't change. This time it kicks out (ref: 408.5609 sec), but I do not know why. Normal operation resumes (for CPU 0), however I think that the processor think all cores are active, and so the limit is 3.2GHz. > 12. recording finished > I hope it is useful. Yes, that was extremely useful.
(In reply to Piotr Kołaczkowski from comment #74) > Kernel 4.1-rc4 did not help. Hmmm... There was one mistake discovered after RC4 was released. It doesn't effect me, but I only use my proposed patch set now. It is late in my time zone, I'll get you to try my patch set tomorrow. > Today I booted laptop on battery once again on kernel 4.1.0-rc4, then > suspended and resumed and found it locked at 800 MHz. Interestingly I > observed that immediately after resume, max frequency was 1.8 GHz and it > changed to 800 MHz a few seconds later, so I guess it was triggered by > switching the governor to powersave by the script. Yes, exactly. Are all CPUs on line after suspend now? You can tell by trying to assign any task to a specific CPU with taskset. It will complain if the CPU is off-line.
> All CPUs are in the trace. What happened in between? I have no idea, I didn't do anything special like suspend/hibernate/reboot. I'll repeat the experiments with tracing 4.1-rc4, but I need to compile your tracing patch in first. > No, frequency stuck at 600 to 700 MHz. So maybe i7z reports them wrong, because it showed all multipliers 8x exactly, and reported frequency 750-800 MHz. For sure, in that state it never exceeded 800 MHz by a single Hz. Next time, besides the perf record, I'll record turbostat output.
(In reply to Piotr Kołaczkowski from comment #77) > So maybe i7z reports them wrong, Yes, I think so. I have seen i7z report incorrect frequencies before. That being said, it is hard to know what information to trust when things are in such a messed up state. TSC seems accurate, and MPERF seems to just track TSC (i.e. CPU seems to be locked in C0 state, regardless of load), but APERF seems lower than the minimum 8X would allow. I had to rebase my proposed patch set for Kernel 4.1. The rebased version can be found at: double u double u double u dot smythies dot com /~doug/linux/intel_pstate/rebase_to_k4.1/ You are already using patch 1 of 5.
First observations from 4.1-rc4 with all your patches 1-5. 1. scaling_governor is fixed on performance and doesn't automatically drop to powersave anymore 2. turboboost was working when booted freshly on AC 3. lost turboboost on the first suspend on battery 4. turboboost doesn't come back even after putting on AC and suspending So far no severe lockup (only missing turboboost), but too early to say for sure. I need to do more experiments.
intel_pstate/no_turbo = 0 intel_pstate/max_perf_pct = 100 intel_pstate/min_perf_pct = 100 Frequency locked at 2400 MHz m4600 intel_pstate # turbostat -d sleep 10 turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org> CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7) CPUID(6): APERF, DTS, PTM, EPB RAPL: 1456 sec. Joule Counter Range, at 45 Watts cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800 8 * 100 = 800 MHz max efficiency 24 * 100 = 2400 MHz TSC frequency cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled) cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7) cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223 32 * 100 = 3200 MHz max turbo 4 active cores 32 * 100 = 3200 MHz max turbo 3 active cores 34 * 100 = 3400 MHz max turbo 2 active cores 35 * 100 = 3500 MHz max turbo 1 active cores cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance) cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.) cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.) cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked) cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled) cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled) cpu0: MSR_PP0_POLICY: 0 cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_PP1_POLICY: 0 cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked) cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled) cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C) cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88270000 (61 C) cpu0: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1) cpu1: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1) cpu2: MSR_IA32_THERM_STATUS: 0x88270000 (61 C +/- 1) cpu3: MSR_IA32_THERM_STATUS: 0x882c0000 (56 C +/- 1) Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 2042 85.24 2396 2396 0 13.58 0.32 0.02 0.84 65 65 0.00 0.00 0.00 0.00 28.93 24.16 0.09 0 0 2211 92.26 2397 2397 1 7.48 0.07 0.01 0.18 62 65 0.00 0.00 0.00 0.00 28.93 24.16 0.09 0 4 1977 82.54 2396 2396 1 17.19 1 1 2007 83.77 2395 2395 1 13.14 0.77 0.05 2.26 63 1 5 1901 79.36 2395 2395 1 17.55 2 2 2018 84.22 2396 2396 1 15.39 0.17 0.00 0.21 65 2 6 2256 94.18 2396 2396 1 5.44 3 3 1873 78.18 2396 2396 1 20.85 0.25 0.00 0.71 59 3 7 2094 87.41 2396 2396 1 11.62 10.005260 sec
(In reply to Piotr Kołaczkowski from comment #79) > First observations from 4.1-rc4 with all your patches 1-5. > > 1. scaling_governor is fixed on performance and doesn't automatically drop > to powersave anymore That is not related to my patch set. And if you are in performance mose, then my patch set is not being used. > 2. turboboost was working when booted freshly on AC > 3. lost turboboost on the first suspend on battery > 4. turboboost doesn't come back even after putting on AC and suspending The initialization mess (or lack thereof) upon exit from suspend is not addressed with my patch set. It will only prevent the one math lock up condition your processor is prone to. By the way, I forgot to mention, I use a different set of default parameter settings now, although it has yet to be decided what we do if the patch set does end up accepted. I'll put a script that sets them all into that web location where I put the patch set.
Ah, ok, that makes sense. Then I need to check why it doesn't do into powersave mode now.
(In reply to Doug Smythies from comment #75) > Executive Summary: > . There are multiple problems. ... > . There might be problems (not sure on this one) where the intel_pstate > driver thinks the new target pstate is the same as previously sent, so it > doesn't send the new one. ... This scenario has now been verified. If one just looks at the code, it looks as though every condition is covered. However, if I just comment out the code in "intel_pstate_set_pstate". i.e. /* if (pstate == cpu->pstate.current_pstate) return; */ Then things work fine upon exit from suspend (using the performance mode method of messing thing up (the "Marien" method from another bug report)). Now I am trying to figure out how it gets into this state: Is it some loophole in the intel_pstate driver? Does it try to set the value too early in the process, and the processor clobbers it itself later? Does some other code or process clobber the value later? While things appear as though turbo is disabled, I actually don't think that is the case, I think it is just that all target pstates somehow get set to the non-turbo maximum.
(In reply to Doug Smythies from comment #83) > (In reply to Doug Smythies from comment #75) > > Executive Summary: > > . There are multiple problems. > ... > > . There might be problems (not sure on this one) where the intel_pstate > > driver thinks the new target pstate is the same as previously sent, so it > > doesn't send the new one. > ... > > This scenario has now been verified. Actually, the logic error is rather glaring, don't know why I didn't see it earlier. I believe the proper way to deal with this issue is with a patch that I just added to another bug report (attachment 177601 [details] of bug 90421).
@Piotr: There are still some things in your trace data, and your comments, that are cause for concern. Could you add the patch referenced in my previous comment (call it patch 6 of 5), and acquire some new trace data.
Yes, I will over the weekend. Currently running kernel 4.1-rc4 with your patches 1-5, and it feels like performance on powersave is lower, but I haven't any hard data to back it up yet. I had a busy week and didn't have much time to dig into that.
Piotr and I have been doing a lot of work in the background on this. The great thing is that Piotr has an absolutely repeatable method for creating the low frequency issue. O.K. here is a status update: 1.) With a reasonable degree of confidence, I claim that the low frequency issue is a Dell BIOS problem. With the current intel_pstate driver, sometimes the CPU frequency locks at less than what should be the minimum for the processor, typically by a factor of between about 0.75 to 0.8. With the intel_pstate driver with my patch set, the frequency does not lock, but is typically (not always) about 0.75 to 0.8 times what is asked for. The earliest posting about this issue that I could find was from late 2012, which pre-dates the existence of the intel_pstate driver. The issue can, and does, come up in mysterious ways. I have trace data from Piotr, which was supposed to be a control sample where everything was fine, but in fact the issue comes and goes a few times. 2.) Issue 1 aside, and as mentioned in my comment 84 above, the intel_pstate driver can lock in such a way that it is out of synchronization with the actual processor target pstate register. The fix patch has been tested and submitted.
I own a Dell E7440, which has an Intel i7-4600U CPU (3.3GHz turbo), and I experience this problem regularly, where the CPU will be locked at very low performance level after resume from suspend (500–600 MHz). It happens seemingly at random and the only way I have found to fix it is to do another suspend, and it will usually be OK on resume again. The kernel uses the 'intel_pstate' driver with 'powersave' governor. Interestingly, Dell recently released a BIOS update for my laptop which claims to address the issue: http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverID=RYJTF&productCode=latitude-e7440-ultrabook Which mentions: > -Fixed CPU may stay at low speed after resume from standby. Unfortunately, the BIOS update did not correct the issue. I am on Ubuntu 14.04 and the 3.16.X-series default kernel.
I've just experienced the same issue with the replacement laptop, which is equipped with an Ivy Bridge i7-3520M. After suspend-to-ram on battery it's frequency is stuck at 900Mhz. The system is running fedora-21 on linux 4.0.4. It is a Dell E6330 (more or less the direct successor to the Sandy Bridge E6320 which led to the creation of the bug-report in the first place).
Created attachment 180021 [details] attachment-28197-0.html Did you test with Doug's patches or without? They fix the problem of getting stuck at a very low frequency for me (not perfectly though, but we think the remaining problem is bios fault). You need to compile kernel from source. 15 cze 2015 20:49 <bugzilla-daemon@bugzilla.kernel.org> napisał(a): > https://bugzilla.kernel.org/show_bug.cgi?id=62851 > > --- Comment #89 from Clemens Eisserer <linuxhippy@gmail.com> --- > I've just experienced the same issue with the replacement laptop, which is > equipped with an Ivy Bridge i7-3520M. After suspend-to-ram on battery it's > frequency is stuck at 900Mhz. The system is running fedora-21 on linux > 4.0.4. > > > It is a Dell E6330 (more or less the direct successor to the Sandy Bridge > E6320 > which led to the creation of the bug-report in the first place). > > -- > You are receiving this mail because: > You are on the CC list for the bug. >
I haven't tested the patches yet, however I reported the issue because this was on an Ivy Bridge based notebook, whereas as far as I have seen, all previous issues were experienced on Sandy Bridge.
@oyvinst: Thanks for your posting. Please also see (and maybe add to): http://en.community.dell.com/support-forums/laptop/f/3518/t/19634759 @Clemens: Thanks for your update postings. What I find intriguing is the 900MHz, which seems to be some new (at least as far as I know) lock up CPU frequency.
Could the people still experiencing low CPU frequencies after suspend please do the following and report back: 1.) (needed once per boot) sudo modprobe msr 2.) before any suspend: sudo rdmsr -a 0x19a 3.) after a suspend that results in the low CPU frequencies: sudo rdmsr -a 0x19a 4.) If the result from step 3 is not 0, then: sudo wrmsr -a 0x19a 0x0 and check it: sudo rdmsr -a 0x19a 5.) Are the CPU frequencies O.K. now?
For people still experiencing low CPU frequencies after suspend: In addition to my request in comment 93, the Intel subject matter expert on thermal interactions and pstates also wants the output from: cd /sys/class/thermal grep -r . *
Just experienced this issue on a Dell Latitude E7240 (Haswell i7-4600U) running linux-4.1.7, cores are stuck between 500-600mhz. The output of /sys/class/thermal is: cooling_device0/type:Processor cooling_device0/power/control:auto cooling_device0/power/async:disabled cooling_device0/power/runtime_enabled:disabled cooling_device0/power/runtime_active_kids:0 cooling_device0/power/runtime_active_time:0 grep: cooling_device0/power/autosuspend_delay_ms: Input/output error cooling_device0/power/runtime_status:unsupported cooling_device0/power/runtime_usage:0 cooling_device0/power/runtime_suspended_time:0 cooling_device0/cur_state:0 cooling_device0/max_state:3 cooling_device1/type:Processor cooling_device1/power/control:auto cooling_device1/power/async:disabled cooling_device1/power/runtime_enabled:disabled cooling_device1/power/runtime_active_kids:0 cooling_device1/power/runtime_active_time:0 grep: cooling_device1/power/autosuspend_delay_ms: Input/output error cooling_device1/power/runtime_status:unsupported cooling_device1/power/runtime_usage:0 cooling_device1/power/runtime_suspended_time:0 cooling_device1/cur_state:0 cooling_device1/max_state:3 cooling_device2/type:Processor cooling_device2/power/control:auto cooling_device2/power/async:disabled cooling_device2/power/runtime_enabled:disabled cooling_device2/power/runtime_active_kids:0 cooling_device2/power/runtime_active_time:0 grep: cooling_device2/power/autosuspend_delay_ms: Input/output error cooling_device2/power/runtime_status:unsupported cooling_device2/power/runtime_usage:0 cooling_device2/power/runtime_suspended_time:0 cooling_device2/cur_state:0 cooling_device2/max_state:3 cooling_device3/type:Processor cooling_device3/power/control:auto cooling_device3/power/async:disabled cooling_device3/power/runtime_enabled:disabled cooling_device3/power/runtime_active_kids:0 cooling_device3/power/runtime_active_time:0 grep: cooling_device3/power/autosuspend_delay_ms: Input/output error cooling_device3/power/runtime_status:unsupported cooling_device3/power/runtime_usage:0 cooling_device3/power/runtime_suspended_time:0 cooling_device3/cur_state:0 cooling_device3/max_state:3 thermal_zone0/mode:enabled thermal_zone0/temp:25000 thermal_zone0/type:acpitz thermal_zone0/power/control:auto thermal_zone0/power/async:disabled thermal_zone0/power/runtime_enabled:disabled thermal_zone0/power/runtime_active_kids:0 thermal_zone0/power/runtime_active_time:0 grep: thermal_zone0/power/autosuspend_delay_ms: Input/output error thermal_zone0/power/runtime_status:unsupported thermal_zone0/power/runtime_usage:0 thermal_zone0/power/runtime_suspended_time:0 thermal_zone0/trip_point_0_temp:107000 thermal_zone0/trip_point_0_type:critical thermal_zone0/policy:step_wise thermal_zone0/passive:0 thermal_zone1/temp:45000 thermal_zone1/type:x86_pkg_temp thermal_zone1/power/control:auto thermal_zone1/power/async:disabled thermal_zone1/power/runtime_enabled:disabled thermal_zone1/power/runtime_active_kids:0 thermal_zone1/power/runtime_active_time:0 grep: thermal_zone1/power/autosuspend_delay_ms: Input/output error thermal_zone1/power/runtime_status:unsupported thermal_zone1/power/runtime_usage:0 thermal_zone1/power/runtime_suspended_time:0 thermal_zone1/trip_point_0_temp:0 thermal_zone1/trip_point_0_type:passive thermal_zone1/trip_point_1_temp:0 thermal_zone1/trip_point_1_type:passive thermal_zone1/policy:step_wise
@Clemens: Please do my comment number 93 stuff. Were you on battery? And was this after a resume from suspend? Note: there is a mistake in step 4. It should be "If the result from step 3 has bit 4 set, then: sudo wrmsr -a 0x19a 0x0 There have been cases where the rdmsr returns non-zero values, where Clock Modulation (bit 4) is not enabled. (In reply to Doug Smythies from comment #93) > Could the people still experiencing low CPU frequencies after suspend please > do the following and report back: > > 1.) (needed once per boot) sudo modprobe msr > 2.) before any suspend: sudo rdmsr -a 0x19a > 3.) after a suspend that results in the low CPU frequencies: > sudo rdmsr -a 0x19a > 4.) If the result from step 3 is not 0, then: > sudo wrmsr -a 0x19a 0x0 > and check it: > sudo rdmsr -a 0x19a > 5.) Are the CPU frequencies O.K. now?
ping clemems.
I haven't experienced this issue for quite a while running fedora stock kernels. As soon as I hit it again, I'll try to perform the steps mentioned in comment 93.
Hmm, okay. I will close this bug for now. Please feel free to re-open it if you can reproduce the problem in vanilla kernel.