Bug 62851

Created attachment 110751 [details]
syslog

Could you try to disable speedstep in the Bios?

From bug 59821, the reporter also meet the same issue and fix by the previous change.
https://bugzilla.kernel.org/show_bug.cgi?id=59821

Disabling speedstep in BIOS seems to help to some degree, however Turbo doesn't kick in anymore (according to powertop the CPU seems to be stuck at it's base-clock of 2.8ghz) :/

A build of one of my java-projects takes now 2:30 instead of 1:30.

So disable speedstep can resolve lowe CPU frequency issue but Turbo mode doesn't work and cpufreq remain 2.8GHz during heavy workload, right?

Please provide the output of "grep . /sys/bus/cpu/devices/cpu*/cpufre/*" and "grep . /sys/devices/system/cpu/cpufreq/*"

Created attachment 112101 [details]
/sys/bus/cpu/devices/

Created attachment 112111 [details]
sys/devices/system/cpu/

Exactly, disabling speedstep helps, but turbo doesn't kick in anymore.

As your grep command didn't work, I simply stacked all the files into archives.

Ping

Hi, could you try the v3.12 kernel. I see some intel_pstate turbo related patches are upstreamed recently.
So current issue became intel_pstate doesn't kick turbo and Cc Dirk.

I just tried the 3.12 kernel on a Dell L502x, and this bug is still present: with speed-step disabled, turbostat reports the cores running at 1.99GHz with 'stress --cpu 8' running. With speed-step enabled, the cores run at 2.56GHz (compared to a turbo max of 2.9GHz).

There is other weird intel_pstate stuff going on - at one point yesterday when I was running kernel 3.12, the laptop had all cores locked at the min frequency (800MHz) no matter what, and this persisted across eight (8) reboots. (This was all with speed-step enabled.) The only thing that restored frequency scaling was rebooting into the 3.11 kernel. After that, it was fine when I rebooted back into 3.12.

Hi Dirk:
       Could you take care this bug? Otherwise, one question: speed-step affects intel_pstate? The symptom seems "yes". Intel_pstate use MSR registers and Bios can involve in?

Can you run "grep . *" in /sys/devices/system/cpu/cpu0/cpufreq with speed_step enabled/disabled and post the results.

If you see the lock up at low frequency again could you do the same thing.

intel_pstate does not save any information across reboots so it feels like the
BIOS is getting in the way somehow.

Also have you seen these issues after suspend/resume cycles?

Created attachment 118151 [details]
output of grep for speedstep enabled/disabled

Here are the results of that grep with speedstep enabled and disabled.

With speedstep disabled, turbo doesn't kick after a suspend/resume cycle, either.

I haven't seen the lock up at low freq again but I'll keep an eye out for it.

I've just seen the lock up at low freq again after a suspend from ram using kernel 3.12.6. turbostat shows each CPU at 0.80 GHz no matter what I run.

Here's the output from the grep command:

affected_cpus:0
cpuinfo_cur_freq:799921
cpuinfo_max_freq:2900000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:2900000
scaling_min_freq:800000
scaling_setspeed:<unsupported>

Same here, after suspend to ram the system was horribly slow.


While the system was affected, running the grep-result yielded an interesting result - instead of beeing stuck at a single low frequency, the frequency oscillated between 612mhz and ~2.2ghz while running "7za b" procuing load on all 4 logical cores:

[root@user-pc cpufreq]# grep . *
affected_cpus:0
cpuinfo_cur_freq:2200515
cpuinfo_max_freq:3500000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:3500000
scaling_min_freq:800000
scaling_setspeed:<unsupported>


[root@user-pc cpufreq]# grep . *
affected_cpus:0
cpuinfo_cur_freq:628578
cpuinfo_max_freq:3500000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:3500000
scaling_min_freq:800000
scaling_setspeed:<unsupported>


[root@user-pc cpufreq]# grep . *
affected_cpus:0
cpuinfo_cur_freq:1170312
cpuinfo_max_freq:3500000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:3500000
scaling_min_freq:800000
scaling_setspeed:<unsupported>


[root@user-pc cpufreq]# grep . *
affected_cpus:0
cpuinfo_cur_freq:629015
cpuinfo_max_freq:3500000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:3500000
scaling_min_freq:800000
scaling_setspeed:<unsupported>


Processing speed is way lower than normal when the machine shows this behaviour:

[ce@user-pc ~]$ 7za b

7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

RAM size:    7865 MB,  # CPU hardware threads:   4
RAM usage:    850 MB,  # Benchmark threads:      4

Dict        Compressing          |        Decompressing
      Speed Usage    R/U Rating  |    Speed Usage    R/U Rating
       KB/s     %   MIPS   MIPS  |     KB/s     %   MIPS   MIPS

22:    3759   291   1254   3657  |    41169   383    970   3714
23:    3607   288   1275   3675  |    34997   363    881   3202
24:    3617   290   1340   3889  |    35905   377    883   3331
25:    3597   292   1408   4107  |    38861   383    953   3654
----------------------------------------------------------------
Avr:          290   1319   3832               377    922   3475
Tot:          333   1121   3653

Could somebody please set the bug status to "New" or "Assigned", please.

As mentioned earlier the problem is still there with 3.12.8, here are the frequency statistics of powertop while running the 7za multithreaded benchmark:

            Package |             Core    |            CPU 0	   CPU 2
                    |                     | Actual	1136 MHz    1116 MHz
Idle         0.3%   | Idle         0.8%   | Idle        29.3%        1.3%
1.71 GHz     3.0%   | 1.71 GHz     3.0%   | 1.71 GHz     2.3%        2.7%
1.60 GHz     5.3%   | 1.60 GHz     4.7%   | 1.60 GHz     3.1%        4.6%
1500 MHz     1.9%   | 1500 MHz     1.9%   | 1500 MHz     1.6%        1.9%
1400 MHz     1.9%   | 1400 MHz     1.9%   | 1400 MHz     1.9%        1.9%
1300 MHz     4.9%   | 1300 MHz     4.9%   | 1300 MHz     3.9%        4.9%
1100 MHz     1.6%   | 1100 MHz     1.6%   | 1100 MHz     1.3%        1.6%
1200 MHz     3.2%   | 1200 MHz     3.2%   | 1200 MHz     1.3%        3.2%
1000 MHz     2.7%   | 1000 MHz     2.7%   | 1000 MHz     1.3%        2.7%
 900 MHz     6.4%   |  900 MHz     6.4%   |  900 MHz     4.5%        6.4%
 800 MHz    33.2%   |  800 MHz    33.2%   |  800 MHz    22.7%       33.2%
2.81 GHz     9.3%   | 2.81 GHz     9.3%   | 2.81 GHz     6.5%        9.3%
3.50 GHz     0.7%   | 3.50 GHz     0.7%   | 3.50 GHz     0.7%        0.7%
2.40 GHz     8.0%   | 2.40 GHz     8.0%   | 3.10 GHz     0.2%        8.0%
3.10 GHz     0.2%   | 3.10 GHz     0.2%   | 2.50 GHz     0.2%        0.2%
2.21 GHz     2.4%   | 2.21 GHz     2.4%   | 1.91 GHz     0.7%        2.4%
2.00 GHz     5.2%   | 2.00 GHz     5.2%   | 2.40 GHz     4.9%        5.2%
2.71 GHz     0.2%   | 2.71 GHz     0.2%   | 2.00 GHz     3.5%        0.2%
1.91 GHz     1.2%   | 1.91 GHz     1.2%   | 2.21 GHz     1.9%        1.2%
2.50 GHz     0.2%   | 2.50 GHz     0.2%   | 2.10 GHz     1.6%        0.2%
2.10 GHz     1.6%   | 2.10 GHz     1.6%   | 3.00 GHz     1.2%        1.6%
3.00 GHz     1.2%   | 3.00 GHz     1.2%   | 2.60 GHz     1.9%        1.2%
2.60 GHz     1.9%   | 2.60 GHz     1.9%   | 2.31 GHz     0.7%        1.9%
2.31 GHz     0.7%   | 2.31 GHz     0.7%   | 1.80 GHz     2.6%        0.7%
1.80 GHz     2.6%   | 1.80 GHz     2.6%   |                          2.6

Does this still happen on the current kernel?

There are comments in this bug report about CPU frequencies locking up at some rate, but it isn't clear how this happens. Somewhere and sometime ago, I mentioned one method. I'll mention it again:

In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the intel_pstate driver:

echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo

All CPU's will now be locked at whatever frequency they were at when the above command was issued.

(In reply to Doug Smythies from comment #18)
> There are comments in this bug report about CPU frequencies locking up at
> some rate, but it isn't clear how this happens. Somewhere and sometime ago,
> I mentioned one method. I'll mention it again:
> 
> In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the
> intel_pstate driver:
> 
> echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
> 
> All CPU's will now be locked at whatever frequency they were at when the
> above command was issued.

I think you mean all cpu' will NOT be locked...?

Clemens and rocko,
please check if the problem still exists in the latest upstream kernel.

Yes, this is still happening on the latest kernel, 3.15-rc8 (3.15.0-031500rc8-generic from the Ubuntu mainline kernel ppa).

Currently my system is sluggish and /proc/cpuinfo indicates that all my cores are stuck at 801.25 MHz.

Here's the output from the earlier grep command:

affected_cpus:0
cpuinfo_cur_freq:801250
cpuinfo_max_freq:2900000
cpuinfo_min_freq:800000
cpuinfo_transition_latency:4294967295
related_cpus:0
scaling_available_governors:performance powersave
scaling_driver:intel_pstate
scaling_governor:performance
scaling_max_freq:2900000
scaling_min_freq:800000
scaling_setspeed:<unsupported>

In case it's relevant, I do have this in the syslog:

Jun  3 00:21:10 trusty-btrfs kernel: [    0.135359] Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, full-width counters, Intel PMU driver.
Jun  3 00:21:10 trusty-btrfs kernel: [    0.135366] perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode
...
Jun  3 08:47:13 trusty-btrfs kernel: [30388.364946] perf interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
...
Jun  4 12:10:17 trusty-btrfs kernel: [126403.543120] perf interrupt took too long (5019 > 5000), lowering kernel.perf_event_max_sample_rate to 25000


This doesn't happen all that often (at least, it didn't with 3.15-rc7 and below) so it would be useful if someone has suggestions for other things to try the next time I see it.

Additional to my comment above, this 'stuck at 800MHz' state persisted across multiple reboots into 3.15-rc8, 3.15-rc7 and 3.14.4 (five in total) using intel_pstate.

I finally rebooted into the default 3.13.0-29-generic kernel and now have some frequency scaling (note that it's using the acpi-cpufreq driver), but the cores won't go beyond 2000 Mhz, ie they won't kick into turbo.

I currently have speedstep enabled in the BIOS.

And additional to both my comments above, I rebooted from 3.13 into 3.15-rc8 and intel_pstate was still stuck at 800MHz. Then I did a powerdown and booted into 3.15-rc8 (without removing the laptop battery) and now things are back to normal. 

I'm still getting the "perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode" message in the syslog.

(In reply to Zhang Rui from comment #19)
> (In reply to Doug Smythies from comment #18)
> > There are comments in this bug report about CPU frequencies locking up at
> > some rate, but it isn't clear how this happens. Somewhere and sometime ago,
> > I mentioned one method. I'll mention it again:
> > 
> > In BIOS disable turbo mode, and boot as normal. Now, disable turbo for the
> > intel_pstate driver:
> > 
> > echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
> > 
> > All CPU's will now be locked at whatever frequency they were at when the
> > above command was issued.
> 
> I think you mean all cpu' will NOT be locked...?
> 

No, I meant to say "locked at whatever frequency", and the issue persists with Kernel 3.15RC7 plus the 4 patches Dirk sent out a few days ago.
Perhaps I should have said "locked at whatever pstate they were in when the command was issued".

Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about 1.78 GHz; Executed the above mentioned turbo off command; Terminated the process that had CPU 7 slightly loaded; Started 12 CPU burning threads; Then:

doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
1800007
1800007
1800007
1800007
1800007
1800007
1800007
1800007
doug@s15:~/temp$ uptime
 08:04:11 up 25 min,  3 users,  load average: 12.06, 10.87, 6.57

(In reply to Doug Smythies from comment #23)
> (In reply to Zhang Rui from comment #19)
> > (In reply to Doug Smythies from comment #18)
> > > There are comments in this bug report about CPU frequencies locking up at
> > > some rate, but it isn't clear how this happens. Somewhere and sometime
> ago,
> > > I mentioned one method. I'll mention it again:
> > > 
> > > In BIOS disable turbo mode, and boot as normal. Now, disable turbo for
> the
> > > intel_pstate driver:
> > > 
> > > echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
> > > 
> > > All CPU's will now be locked at whatever frequency they were at when the
> > > above command was issued.
> > 
> > I think you mean all cpu' will NOT be locked...?
> > 
> 
> No, I meant to say "locked at whatever frequency", and the issue persists
> with Kernel 3.15RC7 plus the 4 patches Dirk sent out a few days ago.
> Perhaps I should have said "locked at whatever pstate they were in when the
> command was issued".
> 
> Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about
> 1.78 GHz; Executed the above mentioned turbo off command; Terminated the
> process that had CPU 7 slightly loaded; Started 12 CPU burning threads; Then:
> 
> doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
> 1800007
> 1800007
> 1800007
> 1800007
> 1800007
> 1800007
> 1800007
> 1800007
> doug@s15:~/temp$ uptime
>  08:04:11 up 25 min,  3 users,  load average: 12.06, 10.87, 6.57
This does not happen when turbo is enabled in the BIOS correct?

When you have the CPU in the locked up condition and the working can you run the command
  rdmsr -x -a 0x199

Turning off turbo in the BIOS is probably not the right answer. I don't know why
the pstate change requests are ignored in this case.

(In reply to Dirk Brandewie from comment #24)
> (In reply to Doug Smythies from comment #23)
...
> > Example: Turbo off in BIOS, then boot; Loaded CPU 7 so as to be at about
> > 1.78 GHz; Executed the above mentioned turbo off command; Terminated the
> > process that had CPU 7 slightly loaded; Started 12 CPU burning threads;
> Then:
> > 
> > doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
> > 1800007
> > 1800007
> > 1800007
> > 1800007
> > 1800007
> > 1800007
> > 1800007
> > 1800007
> > doug@s15:~/temp$ uptime
> >  08:04:11 up 25 min,  3 users,  load average: 12.06, 10.87, 6.57
> This does not happen when turbo is enabled in the BIOS correct?

That is correct, yes.

> 
> When you have the CPU in the locked up condition and the working can you run
> the command
>   rdmsr -x -a 0x199

I get nothing:

doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
2000023
2000023
2000023
2000023
2000023
2000023
2000023
2000023
doug@s15:~/temp$ sudo rdmsr -x -a 0x199
doug@s15:~/temp$ uptime
 11:03:07 up 6 min,  3 users,  load average: 7.81, 3.10, 1.17


> 
> Turning off turbo in the BIOS is probably not the right answer.

Why not? I consider it to be exactly one right way to do it. Regardless, does it actually matter for the discussion? It is something that doesn't work as expected.

> I don't know why the pstate change requests are ignored in this case.

I didn't dig into it much, but here is what I did get:

In this case, the intel_pstate driver never calculates that the target pstate need to increase). Example from "perf record" (every line is similar):

consume  3348 [004]  1933.934957: power:pstate_sample: core_busy=58 scaled=95 state=21 mperf=159893 aperf=94055 freq=2000023

The highest frequency for pstate 21 should be higher, shouldn't it? For the above numbers, and with the default p gain of 20%, the target pstate will never change.

With turbo turned on in the BIOS, "perf record" (starting from no load, so as to capture the transition through where it previously locked up), notice how in state 21, the actual CPU frequency goes higher previously (shown in the next line after the state got set to 21):

           test1  3234 [005]   456.836620: power:pstate_sample: core_busy=55 scaled=90 state=21 mperf=159820 aperf=89279 freq=1899351
           test1  3234 [005]   456.848612: power:pstate_sample: core_busy=61 scaled=91 state=23 mperf=159788 aperf=98671 freq=2099500


Previously, I forgot to load the msr module first. What was actually requested, again:

doug@s15:~/temp$ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
1700000
1700000
1700000
1700000
1700000
1700000
1700000
1700000
doug@s15:~/temp$ uptime
 15:09:57 up 17 min,  3 users,  load average: 9.99, 7.66, 3.78
doug@s15:~/temp$ sudo rdmsr -x -a 0x199
1000
1000
1000
1000
1000
1000
1000
1100

And for that one, "perf record":

           test1  3247 [005]  1127.403413: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159917 aperf=79959 freq=1700000
           test1  3247 [005]  1127.415404: power:pstate_sample: core_busy=50 scaled=89 state=19 mperf=159895 aperf=79947 freq=1700000
           test1  3247 [005]  1127.427395: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159883 aperf=79942 freq=1700000
           test1  3247 [005]  1127.439386: power:pstate_sample: core_busy=50 scaled=89 state=19 mperf=159887 aperf=79943 freq=1700000
           test1  3247 [005]  1127.451380: power:pstate_sample: core_busy=50 scaled=100 state=17 mperf=159912 aperf=79956 freq=1700000
... forever...

This time the driver calculates a new target pstate back, but the frequency never changes.

I have reproduced the lockup Doug is describing on my Acer test system.  Hopefully other vendors BIOS get it right when disabling turbo in the BIOS.

Here is the patch that fixes the issue on my test system based on v3.15-rc8.

commit ecba7c4439f55259911a8f9ee2f57a2636ac120f
Author: Dirk Brandewie <dirk.j.brandewie@intel.com>
Date:   Tue May 20 14:57:21 2014 -0700

    intel_pstate: don't touch turbo bit if trubo disabled in BIOS.
    
    If turbo is disabled in the BIOS bit 38 should be set in
    MSR_IA32_MISC_ENABLE register per section 14.3.2.1 of the SDM Vol 3
    document 325384-050US Feb 2014.  If this bit is set do *not* attempt
    to disable trubo via the MSR_IA32_PERF_CTL register.  On some systems
    trying to disable turbo via MSR_IA32_PERF_CTL will cause subsequent
    writes to MSR_IA32_PERF_CTL not take affect, in fact reading
    MSR_IA32_PERF_CTL will not show the IDA/Turbo DISENGAGE bit(32) as
    set. A write of bit 32 to zero returns to normal operation.
    
    Signed-off-by: Dirk Brandewie <dirk.j.brandewie@intel.com>
---
 drivers/cpufreq/intel_pstate.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c
index eab8ccf..faf5b3a 100644
--- a/drivers/cpufreq/intel_pstate.c
+++ b/drivers/cpufreq/intel_pstate.c
@@ -97,6 +97,7 @@ struct cpudata {
 	struct pstate_data pstate;
 	struct vid_data vid;
 	struct _pid pid;
+	int turbo_disabled;
 
 	u64	prev_aperf;
 	u64	prev_mperf;
@@ -385,7 +386,7 @@ static void byt_set_pstate(struct cpudata *cpudata, int pstate)
 	u32 vid;
 
 	val = pstate << 8;
-	if (limits.no_turbo)
+	if (limits.no_turbo && !cpudata->turbo_disabled)
 		val |= (u64)1 << 32;
 
 	vid_fp = cpudata->vid.min + mul_fp(
@@ -452,7 +453,7 @@ static void core_set_pstate(struct cpudata *cpudata, int pstate)
 	u64 val;
 
 	val = pstate << 8;
-	if (limits.no_turbo)
+	if (limits.no_turbo && !cpudata->turbo_disabled)
 		val |= (u64)1 << 32;
 
 	wrmsrl_on_cpu(cpudata->cpu, MSR_IA32_PERF_CTL, val);
@@ -786,6 +787,7 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy)
 {
 	struct cpudata *cpu;
 	int rc;
+	u64 misc_en;
 
 	rc = intel_pstate_init_cpu(policy->cpu);
 	if (rc)
@@ -793,6 +795,10 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy)
 
 	cpu = all_cpu_data[policy->cpu];
 
+	rdmsrl(MSR_IA32_MISC_ENABLE, misc_en);
+	if (misc_en & MSR_IA32_MISC_ENABLE_TURBO_DISABLE)
+		cpu->turbo_disabled = 1;
+
 	if (!limits.no_turbo &&
 		limits.min_perf_pct == 100 && limits.max_perf_pct == 100)
 		policy->policy = CPUFREQ_POLICY_PERFORMANCE;

updated patch to cover case where the processor is fused to have max and turbo P states as the same and warn if the user tries to enable turbo via sysfs and it is disabled 

commit c35b542da1845cab2d26e1ffc9123457fb256120
Author: Dirk Brandewie <dirk.j.brandewie@intel.com>
Date:   Tue May 20 14:57:21 2014 -0700

    intel_pstate: don't touch turbo bit if turbo disabled or unavailable.
    
    If turbo is disabled in the BIOS bit 38 should be set in
    MSR_IA32_MISC_ENABLE register per section 14.3.2.1 of the SDM Vol 3
    document 325384-050US Feb 2014.  If this bit is set do *not* attempt
    to disable trubo via the MSR_IA32_PERF_CTL register.  On some systems
    trying to disable turbo via MSR_IA32_PERF_CTL will cause subsequent
    writes to MSR_IA32_PERF_CTL not take affect, in fact reading
    MSR_IA32_PERF_CTL will not show the IDA/Turbo DISENGAGE bit(32) as
    set. A write of bit 32 to zero returns to normal operation.
    
    Also deal with the case where the processor does not support
    turbo and the BIOS does not report the fact in MSR_IA32_MISC_ENABLE
    but does report the max and turbo P states as the same value.
    
    Signed-off-by: Dirk Brandewie <dirk.j.brandewie@intel.com>
---
 drivers/cpufreq/intel_pstate.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c
index c034cde..ab554c4 100644
--- a/drivers/cpufreq/intel_pstate.c
+++ b/drivers/cpufreq/intel_pstate.c
@@ -132,6 +132,7 @@ static struct pstate_funcs pstate_funcs;
 
 struct perf_limits {
 	int no_turbo;
+	int turbo_disabled;
 	int max_perf_pct;
 	int min_perf_pct;
 	int32_t max_perf;
@@ -294,7 +295,10 @@ static ssize_t store_no_turbo(struct kobject *a, struct attribute *b,
 	if (ret != 1)
 		return -EINVAL;
 	limits.no_turbo = clamp_t(int, input, 0 , 1);
-
+	if (limits.turbo_disabled) {
+		pr_warn("Turbo disabled by BIOS or unavailable on processor\n");
+		limits.no_turbo = limits.turbo_disabled;
+	}
 	return count;
 }
 
@@ -388,7 +392,7 @@ static void byt_set_pstate(struct cpudata *cpudata, int pstate)
 	u32 vid;
 
 	val = pstate << 8;
-	if (limits.no_turbo)
+	if (limits.no_turbo && !limits.turbo_disabled)
 		val |= (u64)1 << 32;
 
 	vid_fp = cpudata->vid.min + mul_fp(
@@ -455,7 +459,7 @@ static void core_set_pstate(struct cpudata *cpudata, int pstate)
 	u64 val;
 
 	val = pstate << 8;
-	if (limits.no_turbo)
+	if (limits.no_turbo && !limits.turbo_disabled)
 		val |= (u64)1 << 32;
 
 	wrmsrl_on_cpu(cpudata->cpu, MSR_IA32_PERF_CTL, val);
@@ -756,7 +760,7 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy)
 		limits.min_perf = int_tofp(1);
 		limits.max_perf_pct = 100;
 		limits.max_perf = int_tofp(1);
-		limits.no_turbo = 0;
+		limits.no_turbo = limits.turbo_disabled;
 		return 0;
 	}
 	limits.min_perf_pct = (policy->min * 100) / policy->cpuinfo.max_freq;
@@ -799,6 +803,7 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy)
 {
 	struct cpudata *cpu;
 	int rc;
+	u64 misc_en;
 
 	rc = intel_pstate_init_cpu(policy->cpu);
 	if (rc)
@@ -806,8 +811,13 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy)
 
 	cpu = all_cpu_data[policy->cpu];
 
-	if (!limits.no_turbo &&
-		limits.min_perf_pct == 100 && limits.max_perf_pct == 100)
+	rdmsrl(MSR_IA32_MISC_ENABLE, misc_en);
+	if (misc_en & MSR_IA32_MISC_ENABLE_TURBO_DISABLE ||
+		cpu->pstate.max_pstate == cpu->pstate.turbo_pstate) {
+		limits.turbo_disabled = 1;
+		limits.no_turbo = 1;
+	}
+	if (limits.min_perf_pct == 100 && limits.max_perf_pct == 100)
 		policy->policy = CPUFREQ_POLICY_PERFORMANCE;
 	else
 		policy->policy = CPUFREQ_POLICY_POWERSAVE;

My test computer motherboard (with i7-2600K CPU @ 3.40GHz) is Asus, and I last updated the BIOS on 2013.11.15.

I tried, and confirm, Dirk's patch (version 2). Thanks and sorry for the slight diversion on this bug report.

Is my issue (cores sometimes get stuck at 800MHz using pstate even after reboot) related to this bug?

The issue *might* be related to the power cord. The last time this happened, a complete shutdown and reboot didn't fix the issue, but removing and replugging the power cord and then a complete shutdown and reboot did fix it.

Rocko are you still having this issue?  The patch above has been merged.

No, I haven't seen the cores getting stuck at low frequency in a long time now, IIRC not since I reported it in June (comment 29). For info, I've been using 3.16.0/1/2/3 since each one came out and have just switched to 3.17-rc6.

Unfourtunately, I just got stuck with 620mhz after resume-from-ram with linux-3.18.3, so I guess it still hasn't been fixed.

Ok, this doesn seem to be strictly linked with suspend-to-ram - I just experienced all cores stuck at 625mhz after 4h of uptime (cold boot) without any suspend before.

The only interesting entries in syslog are:

[11652.186969] perf interrupt took too long (2558 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[14809.974491] dell_wmi: Received unknown WMI event (0x12)

Hi Clemens,
Is it possible you are experiencing thermal issues?  Can you please try to duplicate with a vanilla upstream kernel and send me the exact duplication steps?

Thanks.

I am 100% sure those issue were not triggered by thermal issues - the system showed the behaviour after suspend from ram - where the CPU sitting withing idle without power for hours. Also, there were not the usual thermal messages which showed up when loading all 4 cores for hours (during FPGA synthesis).

However, as the notebook dies a few weeks ago, I can't test it anymore :/

I have Dell Precision M4600 laptop and have a similar issue.
The processor is 4-core i7 Sandy Bridge:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
stepping	: 7
microcode	: 0x29
cpu MHz		: 803.906
cache size	: 6144 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt
bugs		:
bogomips	: 4789.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:
 
After the laptop is freshly booted from poweroff, full CPU frequency range from 800 MHz to 2.4+ GHz (including turbo boost) is used, depending on load. 

After I suspend to RAM and resume on AC power, everything is almost ok, almost full frequency range used, with the exception of turbo-boost. 

After I suspend to RAM on AC power or battery and *resume on battery*, all cores of the processor are stuck at 800 MHz. This behavior is repeatable - it happens always on resuming on battery and never happened to me on resuming on AC.

Upgrade to recent BIOS A16 for this model did not help.

Somewhere on the Internet I found a workaround to overwrite /sys/devices/system/cpu/intel_pstate/max_perf_pct file twice, once to change the value for something lower than 100, then to bring it back to 100 and it unlocked full frequency range. 

I observed the problem on only some kernel versions (I haven't tried all of them but):

3.16.x - buggy, but workaround works
3.18.12 - ok (I was so happy to see it fixed)
3.19.4 to 3.19.8 - all seem ok (currently I'm running 3.19.8)
4.0.1, 4.0.2 - buggy again (regression?), and the workaround seem to not work anymore, but I haven't tried hard 

I'm using Linux Mint 17.1 and taking kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/

For sure those issues are not thermal- nor CPU load-related for me.

When it happens, lm-sensors reports temperatures well below 50 C for all cores and the fans are either off (total silence) or at the first speed step. Downscaling and locking frequency to 800 MHz never happens when I load the laptop heavily with multithreaded tasks - if I do it when the bug is not effective and full range is available, the CPU frequency rather goes up to 3.2 GHz for all cores (turbo-boost), fans go to full speed, but it never overheats. I clean fans/heatsinks from dust regularly ;)

My current sensors output when I'm writing this:
acpitz-virtual-0
Adapter: Virtual device
temp1:        +25.0°C  (crit = +107.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +43.0°C  (high = +86.0°C, crit = +100.0°C)
Core 0:         +43.0°C  (high = +86.0°C, crit = +100.0°C)
Core 1:         +39.0°C  (high = +86.0°C, crit = +100.0°C)
Core 2:         +41.0°C  (high = +86.0°C, crit = +100.0°C)
Core 3:         +37.0°C  (high = +86.0°C, crit = +100.0°C)

i8k-virtual-0
Adapter: Virtual device
fan1:        2571 RPM
fan2:        2609 RPM
temp1:        +42.0°C  
temp2:        +39.0°C  
temp3:        +38.0°C  
temp4:        +38.0°C 

For me the problem mentioned here is suspend/resume related.

@Piotr:

Could you please supply a little additional information: Both before and after suspend, the output from:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
cat /sys/devices/system/cpu/intel_pstate/no_turbo
cat /sys/devices/system/cpu/intel_pstate/max_perf_pct
cat /sys/devices/system/cpu/intel_pstate/min_perf_pct

Also the before and after output from:

sudo turbostat -d sleep 10

Do you use the generic or low latency kernels from the PPA? (I assume 64 bit).

Thanks for fast reply.
I use generic ones. 
I'll reboot to 4.0.2 and post the results.

I don't have turbostat - where to get linux-tools package for kernel 4.0.2?
I managed to get a creenshot of i7z, though.

Fresh boot on AC:

pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo
0
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct
100
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct
22


Cpu speed from cpuinfo 2394.00Mhz
cpuinfo might be wrong if cpufreq is enabled. To guess correctly try estimating via tsc
Linux's inbuilt cpu_khz code emulated now
True Frequency (without accounting Turbo) 2394 MHz
  CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz

Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  35x/34x/32x/32x
  Real Current Frequency 2045.84 MHz [99.75 x 20.51] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %   C7 %  Temp
        Core 1 [0]:       2045.84 (20.51x)      10.8    38.8       1       0      51    60
        Core 2 [1]:       2038.05 (20.43x)      4.26    10.6    2.02       0    83.8    60
        Core 3 [2]:       1399.97 (14.03x)      2.72     2.5       1       0    94.9    60
        Core 4 [3]:       1513.23 (15.17x)      2.05    1.79       1       0    95.9    60



C0 = Processor running without halting
C1 = Processor running with halts (States >C0 are power saver)
C3 = Cores running with PLL turned off and core cache turned off
C6 = Everything in C3 + core state saved to last level cache
  Above values in table are in percentage over the last 1 sec
[core-id] refers to core-id number in /proc/cpuinfo
'Garbage Values' message printed when garbage values are read
  Ctrl+C to exit


After unplugging:

pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo
0
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct
100
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct
22

Cpu speed from cpuinfo 2394.00Mhz
cpuinfo might be wrong if cpufreq is enabled. To guess correctly t
Linux's inbuilt cpu_khz code emulated now
True Frequency (without accounting Turbo) 2394 MHz
  CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz

Socket [0] - [physical cores=4, logical cores=8, max online cores 
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  35x/34x
  Real Current Frequency 1411.38 MHz [99.75 x 14.15] (Max of below
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)% 
        Core 1 [0]:       1411.38 (14.15x)      9.45    30.2      
        Core 2 [1]:       1175.90 (11.79x)      6.05    5.15      
        Core 3 [2]:       1132.88 (11.36x)      6.97    6.57      
        Core 4 [3]:       1203.24 (12.06x)      3.72    3.97      



C0 = Processor running without halting
C1 = Processor running with halts (States >C0 are power saver)
C3 = Cores running with PLL turned off and core cache turned off
C6 = Everything in C3 + core state saved to last level cache
  Above values in table are in percentage over the last 1 sec
[core-id] refers to core-id number in /proc/cpuinfo
'Garbage Values' message printed when garbage values are read
  Ctrl+C to exit



AFTER suspend/resume.
I must take back what I said before - this doesn't seem to happen always. I had to try twice to trigger the problem.

pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/no_turbo
0
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct
100
pkolaczk@m4600 ~ $ cat /sys/devices/system/cpu/intel_pstate/min_perf_pct
22

Linux's inbuilt cpu_khz code emulated now
True Frequency (without accounting Turbo) 2394 MHz
  CPU Multiplier 24x || Bus clock frequency (BCLK) 99.75 MHz

Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 2493.75 MHz (99.75 x [25])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  35x/34x/32x/32x
  Real Current Frequency 798.04 MHz [99.75 x 8.00] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %   C7 %  Temp
        Core 1 [0]:       797.83 (8.00x)        3.71    3.18       1       0    94.6    59
        Core 2 [1]:       797.97 (8.00x)        3.44    4.35       1       0    93.5    58
        Core 3 [2]:       797.86 (8.00x)        6.43    10.7       0       0    87.2    58
        Core 4 [3]:       798.04 (8.00x)        1.15    1.36       0       0    98.3    58


The multiplier is stuck at 8.00x and never changes.

(In reply to Piotr Kołaczkowski from comment #36)
> I observed the problem on only some kernel versions (I haven't tried all of
> them but):
> 
> 3.16.x - buggy, but workaround works
> 3.18.12 - ok (I was so happy to see it fixed)
> 3.19.4 to 3.19.8 - all seem ok (currently I'm running 3.19.8)
> 4.0.1, 4.0.2 - buggy again (regression?), and the workaround seem to not
> work anymore, but I haven't tried hard 

That is useful thanks.

Resume from suspend is not working at all for me with kernels 4.1RC1 or RC2 or RC3. Upon resume, only CPU 0 comes back on-line.

It works fine for me with Kernel 4.0. after resume from suspend, turbo works as expected as does the normal frequency verses load curve.

Shortly, I will look at the post added while I was doing this one, however I do not like i7z and do not trust what is ways.

If you give me some hints how to get linux-tools installed on my system, I can try with turbostat. Is there some special ppa where I an install it from, or can I get the sources somewhere and compile?

i7z frequency looks correct - when it is stuck at 800 MHz, the laptop is visibly much slower. Particularly CPU heavy tasks like compilation take "forever".

"It works fine for me with Kernel 4.0. after resume from suspend, turbo works as expected as does the normal frequency verses load curve"

For me the critical thing triggering this is resuming on battery. If I resume on AC power, everything is ok.

I recall having problems similar to that with some much earlier kernels < 3.10, which didn't use intel_pstate. But that was long time ago when I was using Ubuntu and I can't reproduce that setup anymore. Maybe this particular hardware or BIOS in this laptop has some weird bug in it that shows up with only some kernels?

(In reply to Piotr Kołaczkowski from comment #42)
> If you give me some hints how to get linux-tools installed on my system, I
> can try with turbostat. Is there some special ppa where I an install it
> from, or can I get the sources somewhere and compile?

turbostat source is included in the kernel source (tools/power/x86/turbostat/turbostat.c), you just have to go to its directory and build it.

I did e-mail you a link to where I have a recently compiled turbostat on my web site.

Currently, I am bisecting the kernel to determine where suspend broke for my system. However, for every "bad" I forget to re-boot, and then only have one working CPU for the next compile.

This is a bit of a tangent, but:

(In reply to Doug Smythies from comment #41)

> Resume from suspend is not working at all for me with kernels 4.1RC1 or RC2
> or RC3. Upon resume, only CPU 0 comes back on-line.

For my system, suspend was broken by:

3c18d447b3b36a8d3c90dc37dfbd363cdb685d0a "sched/core: Check for available DL bandwidth in cpuset_cpu_inactive()"

And is fixed by:

533445c6e533 "sched/core: Fix regression in cpuset_cpu_inactive() for suspend"

which I think, but am not sure, is queued for Kernel 4.1RC4.

(In reply to Piotr Kołaczkowski from comment #40)

> AFTER suspend/resume.
> I must take back what I said before - this doesn't seem to happen always. I
> had to try twice to trigger the problem.

Suggesting some sort of race condition or other timing sensitive event, which might explain why, or so it seems, some processors do not experience this issue, but some do.

Today I installed kernel 4.0.3 and compiled turbostat for it.

So far, after more than 10 suspends/resumes, both on AC or battery, everything works fine including turbo-boost (frequencies go above 2.4 GHz under load, so it must be kicking in).

However, now I have disabled Optimus in BIOS and I'm running on Intel Video driver. 

Previous problems happened when I had Optimus on nvidia-prime was in mode "nvidia", not "intel".

I'll reboot to Optimus and do the same experiment with multiple suspend/resume.

Just for the record: my lattitide E6320 didn't have Optimus, only the integrated intel hd graphics.

Ah, I messed up my previous post a little bit.
I had Optimus *enabled* and I was running on Intel Video driver, i.e. prime-select intel. It is not possible to run this laptop on Intel card, without turning on optimus.

I found something very interesting.
I almost thought everything was fine, but after several suspend/resumes on kernel 4.0.3 I noticed now my CPU is capped at multiplier 18x, that is 1800 MHz. Much better than the previous 800 MHz, but still far from perfect.

turbostate before suspend (but CPU was not loaded):


turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7)
CPUID(6): APERF, DTS, PTM, EPB
RAPL: 1456 sec. Joule Counter Range, at 45 Watts
cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800
8 * 100 = 800 MHz max efficiency
24 * 100 = 2400 MHz TSC frequency
cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled)
cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223
32 * 100 = 3200 MHz max turbo 4 active cores
32 * 100 = 3200 MHz max turbo 3 active cores
34 * 100 = 3400 MHz max turbo 2 active cores
35 * 100 = 3500 MHz max turbo 1 active cores
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked)
cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x882d0000 (55 C)
cpu0: MSR_IA32_THERM_STATUS: 0x882d0000 (55 C +/- 1)
cpu1: MSR_IA32_THERM_STATUS: 0x882f0000 (53 C +/- 1)
cpu2: MSR_IA32_THERM_STATUS: 0x882f0000 (53 C +/- 1)
cpu3: MSR_IA32_THERM_STATUS: 0x88340000 (48 C +/- 1)
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt
       -       -       8    0.97     819    2395       0    1.74    0.02    0.00   97.27      55      55    5.01    0.04    5.46   80.17    4.01    0.33    0.09
       0       0      16    1.98     820    2395       1    1.02    0.06    0.00   96.94      55      55    5.01    0.04    5.46   80.17    4.01    0.33    0.09
       0       4       0    0.05     808    2395       1    2.94
       1       1      20    2.43     826    2395       1    0.69    0.00    0.00   96.88      54
       1       5       1    0.07     835    2395       1    3.05
       2       2      11    1.35     820    2395       1    0.56    0.01    0.00   98.08      54
       2       6       0    0.02     806    2395       1    1.89
       3       3      10    1.24     810    2395       1    1.58    0.02    0.00   97.16      54
       3       7       5    0.60     802    2395       1    2.22
10.003878 sec


After resume on battery:
turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7)
CPUID(6): APERF, DTS, PTM, EPB
RAPL: 1456 sec. Joule Counter Range, at 45 Watts
cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800
8 * 100 = 800 MHz max efficiency
24 * 100 = 2400 MHz TSC frequency
cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled)
cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223
32 * 100 = 3200 MHz max turbo 4 active cores
32 * 100 = 3200 MHz max turbo 3 active cores
34 * 100 = 3400 MHz max turbo 2 active cores
35 * 100 = 3500 MHz max turbo 1 active cores
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked)
cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x881f0000 (69 C)
cpu0: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1)
cpu1: MSR_IA32_THERM_STATUS: 0x88200000 (68 C +/- 1)
cpu2: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1)
cpu3: MSR_IA32_THERM_STATUS: 0x88240000 (64 C +/- 1)
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt
       -       -    1138   63.41    1795    2395       0   20.20    1.71    0.14   14.54      70      70    0.19    0.15    0.10    0.02   17.13   12.87    0.09
       0       0    1457   81.20    1794    2395       0   12.96    1.46    0.18    4.20      70      70    0.19    0.15    0.10    0.02   17.13   12.87    0.09
       0       4     959   53.40    1795    2395       0   40.76
       1       1    1170   65.17    1796    2395       0   18.63    1.65    0.25   14.30      67
       1       5    1078   60.01    1796    2395       0   23.79
       2       2    1155   64.33    1795    2395       0   14.90    1.96    0.07   18.74      67
       2       6    1186   66.03    1795    2395       0   13.20
       3       3    1115   62.11    1795    2395       0   15.16    1.76    0.04   20.94      60
       3       7     988   55.05    1795    2395       0   22.22
10.002551 sec

And state of sensors (machine under load, compiling some project in background):

acpitz-virtual-0
Adapter: Virtual device
temp1:        +25.0°C  (crit = +107.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +65.0°C  (high = +86.0°C, crit = +100.0°C)
Core 0:         +65.0°C  (high = +86.0°C, crit = +100.0°C)
Core 1:         +62.0°C  (high = +86.0°C, crit = +100.0°C)
Core 2:         +61.0°C  (high = +86.0°C, crit = +100.0°C)
Core 3:         +56.0°C  (high = +86.0°C, crit = +100.0°C)

i8k-virtual-0
Adapter: Virtual device
Processor Fan:    0 RPM
Video Fan:        0 RPM
CPU:            +64.0°C  
Ambient:        +43.0°C  
SODIMM:         +44.0°C  
SODIMM:         +44.0°C

Putting the laptop back on AC, without reboot, *does not unlock it*.

Putting the laptop back on AC and suspending / resuming unlocks it partially. The maximum frequency is 2.4 GHz, so no turbo.

turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7)
CPUID(6): APERF, DTS, PTM, EPB
RAPL: 1456 sec. Joule Counter Range, at 45 Watts
cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800
8 * 100 = 800 MHz max efficiency
24 * 100 = 2400 MHz TSC frequency
cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled)
cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223
32 * 100 = 3200 MHz max turbo 4 active cores
32 * 100 = 3200 MHz max turbo 3 active cores
34 * 100 = 3400 MHz max turbo 2 active cores
35 * 100 = 3500 MHz max turbo 1 active cores
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked)
cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x881f0000 (69 C)
cpu0: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1)
cpu1: MSR_IA32_THERM_STATUS: 0x88240000 (64 C +/- 1)
cpu2: MSR_IA32_THERM_STATUS: 0x881f0000 (69 C +/- 1)
cpu3: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1)
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt
       -       -     622   25.95    2399    2399       0   24.27    2.16    0.00   47.63      69      69    0.00    0.00    0.00    0.00   16.17   12.78    0.09
       0       0     681   28.35    2402    2402       1   21.96    0.50    0.00   49.19      65      69    0.00    0.00    0.00    0.00   16.17   12.78    0.09
       0       4     600   24.98    2400    2400       1   25.29
       1       1     593   24.72    2400    2400       1   48.25    2.06    0.00   24.97      69
       1       5    1234   51.40    2400    2400       1   21.57
       2       2     687   28.66    2398    2398       1   16.72    3.09    0.00   51.53      65
       2       6     392   16.35    2397    2397       1   29.01
       3       3     551   22.99    2397    2397       1    9.24    2.99    0.00   64.78      58
       3       7     242   10.09    2395    2395       1   22.06
10.002048 sec

Resuming again on battery brought it back to 1.8 GHz limit.
I repeated that 6 times and the maximum achievable frequency was always 1.8 GHz.

Then I resumed again on AC 6 times, and all the times the limit was 2.4 GHz.


The good news is that so far I *never* got 800 MHz limit with 4.0.3 kernel, despite suspending/resuming more than 25 times since last reboot. On 4.0.2 that condition was pretty easy to trigger. 

Also, I'm pretty impressed with the overall stability of the suspend/resume process compared to 3.18.x and 3.19.x series. With those earlier kernels there was some weird intermittent bug preventing my laptop going to sleep sometimes. It looked as if it went to sleep fine (including turning off display, downspinning hdd, turning off fans, turning off wireless / bt) but immediately resumed afterwards (and funny - it was fully operational then, so I could just hit "suspend" once again, and often it did that fine). I also had a permanent freeze once after resuming on 3.19.x. So far on 4.0.3 none of those problems ever happened, so I'm now staying on that kernel and will observe it.

turbostat after a fresh reboot, this time under load.
This is how it should look like :)

turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7)
CPUID(6): APERF, DTS, PTM, EPB
RAPL: 1456 sec. Joule Counter Range, at 45 Watts
cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800
8 * 100 = 800 MHz max efficiency
24 * 100 = 2400 MHz TSC frequency
cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled)
cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223
32 * 100 = 3200 MHz max turbo 4 active cores
32 * 100 = 3200 MHz max turbo 3 active cores
34 * 100 = 3400 MHz max turbo 2 active cores
35 * 100 = 3500 MHz max turbo 1 active cores
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked)
cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88110000 (83 C)
cpu0: MSR_IA32_THERM_STATUS: 0x88150000 (79 C +/- 1)
cpu1: MSR_IA32_THERM_STATUS: 0x88110000 (83 C +/- 1)
cpu2: MSR_IA32_THERM_STATUS: 0x88120000 (82 C +/- 1)
cpu3: MSR_IA32_THERM_STATUS: 0x88170000 (77 C +/- 1)
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt
       -       -    2159   68.07    3172    2398       0   24.47    0.48    0.03    6.95      86      86    0.04    0.06    0.00    0.00   50.02   45.72    0.09
       0       0    2636   82.94    3178    2400       1   12.28    0.93    0.12    3.73      82      86    0.04    0.06    0.00    0.00   50.02   45.72    0.09
       0       4    1800   56.66    3177    2400       1   38.56
       1       1    2558   80.50    3178    2400       1   17.60    0.03    0.00    1.88      86
       1       5    2446   77.01    3176    2398       1   21.09
       2       2    1960   61.82    3170    2398       1   26.40    0.66    0.00   11.12      78
       2       6    2266   71.57    3166    2397       1   16.64
       3       3    2140   67.53    3169    2397       1   21.08    0.32    0.00   11.06      83
       3       7    1471   46.51    3162    2395       1   42.09

Now, all on AC, I suspended/resumed multiple times (over 5x) and turbo-boost always worked (checked under load - frequencies about 3.2 GHz reported on all cores).

This is really weird.
Now I suspended/resumed multiple times on battery, and turboboost works fine.

So I still have no idea what triggers this. However, when the laptop goes into that "wrong" state, it becomes very repeatable: maximums are 2.4 GHz on AC and 1.8 GHz on battery and it remembers its state until the next reboot.

Any ideas how to debug it further? Is it possible to enable more logging?

I did another reboot on AC, tried doing multiple suspends both on AC and battery, and had no luck reproducing it (still kernel 4.0.3). 

Then I booted the laptop on battery power. Turboboost was working fresh after reboot. Then I suspended it twice on battery, and after the second resume I have my laptop again in "invalid state" with limit 1.8 GHz. :/

Both min_perf_pct and max_perf_pct are 100.
no_turbo = 0.

I tried the old workaround trick with overwriting the contents of max_perf_pct.
I issued the following:

m4600 intel_pstate # echo 99 >> max_perf_pct
// waited a few seconds and then
m4600 intel_pstate # echo 100 >> max_perf_pct

Surprising, now my max frequency is... 2.6 GHz, multiplier 26x. No suspend/resume was needed to unlock it, AC was never connected in this session since reboot, still working on battery.

Connected to AC, suspended / resumed. Max freq was at 2.4 GHz (so lower than previous 2.6 GHz on battery). Then performed the same "trick" with max_perf_pct and it unlocked the freq range to at least 3.2 GHz.

> Both min_perf_pct and max_perf_pct are 100.

Huh? Are you somehow using the performance mode governor now?
Do you know how min_perf_pct got to 100?

These are potentially significant:

> cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
> cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance)

> cpu0: MSR_IA32_POWER_CTL: 0x0004005d (C1E auto-promotion: DISabled)
> cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled)

Are you comfortable compiling the kernel?
It might be time to acquire some trace data,
but a patch (attachment 169891 [details]) is needed.

On my system (no tlp, no thermald, no battery) the MSR_IA32_ENERGY_PERF_BIAS setting is not restored upon resume from suspend. It is "balanced" before the suspend and "performance" after. Kernel 4.1rc3 + 533445c6e533.

I have no tlp or any other extra pm packages installed.

I booted to older 4.0.2 kernel (on AC) and observed that:
Initially after logging in I got:

m4600 intel_pstate # cat min_perf_pct 
100
m4600 intel_pstate # cat max_perf_pct 
100

But some time later (a minute?), when double-checking I got:

m4600 intel_pstate # cat max_perf_pct 
100
m4600 intel_pstate # cat min_perf_pct 
22

I didn't do *anything* to manually change those settings.

> Are you comfortable compiling the kernel?
It might be time to acquire some trace data,
but a patch (attachment 169891 [details]) is needed.

I did that last time about 10 years ago. Sounds fun. No problem. :)

So please confirm this plan is good:
1. download sources from kernel.org
2a. apply three Ubuntu patches from ppa/mainline, for 4.0.3 kernel version
2b. apply any other patches from you
3. make oldconfig
4. make
5. somehow install new kernel to mint - I have no idea how one does that - any hints? make install + update-grub? Or something else?

Fresh reboot, immediately after logging into the graphical session, I issued the following commands:

pkolaczk@m4600 ~ $ sudo su
[sudo] password for pkolaczk: 
m4600 pkolaczk # cd /sys/devices/system/cpu/intel_pstate/
m4600 intel_pstate # while true; do cat min_perf_pct; sleep 1; done
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
22
22
22
22
22
22
22
22


Leaving it running and then suspending/resuming on AC twice:
22
22
22
22
22
22
100
100
100
22
22
22
22
22
22
22
22
22
22
22
22
22
22
22
100
100
100
22
22
22
22

Is it expected, or is something in Ubuntu/Mint changing governors?
What to look for?

I also checked I *don't* have thermald installed.

(In reply to Piotr Kołaczkowski from comment #63)
> Fresh reboot, immediately after logging into the graphical session, I issued
> the following commands:
>
> Is it expected, or is something in Ubuntu/Mint changing governors?
> What to look for?

Yes, There is a script that is run on startup that sleeps for 1 minute, then sets "ondemand" mode if it exits, or "powersave" mode if it exists.
see /etc/init.d/ondemand (at least I think for your distro).

How it gets restored after suspend I do not know.

Oh yes, you're right. There is a script that does exactly that. I'll disable it temporarily and retry the experiments. In the meantime, I'm compiling a patched kernel 4.0.3 with more tracing.

(In reply to Piotr Kołaczkowski from comment #66)
> Oh yes, you're right. There is a script that does exactly that. I'll disable
> it temporarily and retry the experiments. In the meantime, I'm compiling a
> patched kernel 4.0.3 with more tracing.

You should leave the script enabled, as you want "powersave" mode (or set it manually). Once you have the patched kernel, create your failure scenario and run:

sudo ~/bin/perf record -a --event=power:pstate_sample sleep 300

in the 5 minute sample time do whatever loading of the CPU that shows the issue and do the thing that unlocks it or whatever (adjust the 300 seconds to longer if needed). Either post the resulting pref.data here or e-mail it to me directly, and I'll post process it.

Sounds good. In the meantime, which parts of the kernel sources should I look into to get better understanding of intel_pstate driver? Can you point me to some papers / documentation on this? I'm eager to try out any patches when you have them, but I'd like to also get some better understanding how it all works :)

(In reply to Piotr Kołaczkowski from comment #68)
> Sounds good. In the meantime, which parts of the kernel sources should I
> look into to get better understanding of intel_pstate driver?

drivers/cpufreq/intel_pstate.c

> Can you point
> me to some papers / documentation on this?

Documentation/cpu-freq/intel-pstate.txt

When building kernel with 
fakeroot make-kpkg --initrd kernel_image kernel_headers modules_image

I ran into this bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745686

restore_upstream_debianization
test ! -f scripts/package/builddeb.kpkg-dist ||	mv -f scripts/package/builddeb.kpkg-dist scripts/package/builddeb
test ! -f scripts/package/Makefile.kpkg-dist ||	mv -f scripts/package/Makefile.kpkg-dist scripts/package/Makefile
/usr/bin/make  INSTALL_MOD_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3	     \
		INSTALL_FW_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3/lib/firmware/4.0.3  \
		INSTALL_PATH=/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot  install
make[2]: Wejście do katalogu `/home/pkolaczk/Pobrane/linux-4.0.3'
scripts/kconfig/conf --silentoldconfig Kconfig
make[2]: Opuszczenie katalogu `/home/pkolaczk/Pobrane/linux-4.0.3'
make[2]: Wejście do katalogu `/home/pkolaczk/Pobrane/linux-4.0.3'
sh ./arch/x86/boot/install.sh 4.0.3 arch/x86/boot/bzImage \
		System.map "/home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot"
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.0.3 /home/pkolaczk/Pobrane/linux-4.0.3/debian/linux-image-4.0.3//boot/vmlinuz-4.0.3
/etc/kernel/postinst.d/apt-auto-removal: 84: /etc/kernel/postinst.d/apt-auto-removal: cannot create /etc/apt/apt.conf.d//01autoremove-kernels.dpkg-new: Permission denied

Any quick workaround?

A lot of the published methods for compiling the kernel have issues.

Here is how I compile the kernel (from the linux directory of the source):

1.) steal the ubuntu kernel configuration:

cp /boot/config-4.1.0-040100rc3-generic .config

2.) disable excessive debug settings in ubuntu config:

scripts/config --disable DEBUG_INFO

3.) Compile the kernel, with a localized name:
(I always time the compile, and I have 8 CPUs.)

time make -j8 deb-pkg LOCALVERSION=-fix

4.) Install:

sudo dpkg -i ../linux-headers-4.1.0-rc3-fix_4.1.0-rc3-fix-33_amd64.deb
sudo dpkg -i ../linux-image-4.1.0-rc3-fix_4.1.0-rc3-fix-33_amd64.deb

5.) Re-boot and select the desired kernel from the grub menu (I always have grub set to display its menu for about 20 seconds).

Thanks, that did it! I'm writing this from my new kernel. Will ping you soon when I have some good perf.data.

(In reply to Doug Smythies from comment #60)
> On my system (no tlp, no thermald, no battery) the MSR_IA32_ENERGY_PERF_BIAS
> setting is not restored upon resume from suspend. It is "balanced" before
> the suspend and "performance" after. Kernel 4.1rc3 + 533445c6e533.

Addendum 1: Only for CPU 0. The other CPUs are fine.
Addendum 2: Same as of Kernel 4.1RC4.

Kernel 4.1-rc4 did not help.
Today I booted laptop on battery once again on kernel 4.1.0-rc4, then suspended and resumed and found it locked at 800 MHz. Interestingly I observed that immediately after resume, max frequency was 1.8 GHz and it changed to 800 MHz a few seconds later, so I guess it was triggered by switching the governor to powersave by the script.

Everybody: Piotr sent me some trace data via e-mail.

Executive Summary:
. There are multiple problems.
. There are problems with math lockup within the intel_pstate driver.
. There are problems with processor configuration upon exit from suspend.
. There might be problems (not sure on this one) where the intel_pstate driver thinks the new target pstate is the same as previously sent, so it doesn't send the new one. However in the meantime there has been a suspend and the target pstate is not what the driver thinks it is.
. Apparently the processor can get into a state where the pstate isn't supposed to even be possible.

Suggestions:
. Math lockup: Use my proposed patch set instead. The math can not lock up.
. Processor Configuration: We should synchronize ourselves with kernel 4.1RC4. There was a bunch of activity as recently as Monday morning on this, with RC4 delayed to later on Monday as a result. However, I am aware of one mistake, that I guess will carry over to RC5.
. Confused intel_pstate driver: Somehow it needs to re-initialize properly, which may or may not occur as a result of fixing the above.

Details: (with Piotr's e-mail copied herein):

> I managed to reproduce the problem.
> This time it locked at 800 MHz for the first time (!)
> I suspended it.

This is a math problem within the current version of the intel_pstate driver.
For your particular processor, and for the default settings used, it will never break out of a target pstate of 8 under the given conditions of your trace.

> perf1.data was recorded in the following session:
> 1. fresh boot on battery
> 2. started perf recording
> 3. started compilation of some big project to load CPU;
> got freq reported by i7z up to 3.2 GHz
> 4. suspended
> 5. resumed, frequency locked at 800 MHz

The trace data reveals:
. Only CPU 0 came back on-line. This is the same problem I had, and is fixed in kernel 4.1RC4
. MPERF, APERF, and TSC are ridiculous after the suspend. Obvioulsy, the 64 bit counters reset during the suspend. This situation may correct itself if the intel_pstate driver is re-initialized properly.
. CPU 0 seems to be in performance mode for a few seconds afterwards. However the CPU frequency suggests the target pstate is 18. I suspect the intel_pstate driver never sends the new target pstate because it thinks it is unchanged (35 was the last one sent before the suspend).
. The frequency is not locked at 800 MHz. After a number of seconds it ends up in the 600 to 700 Mhz range. (???)
. I assume that at this point the governor was set back to powersave.
. at some point performance mode is set again, and the CPU frequency does respond, but it doesn't make much sense. Eventually it does settle to 2.6 GHz.
. Then powersave again. And 600 to 700 Mhz again. There is a perturbation that causes a 1 sample increase in frequency to 853 MHz.

> 6. tried doing the echo 99 >> max_perf_pct trick a few times,
> but it didn't help this time

No, for this particular lockup scenario, it wouldn't.

> 7. recording finished

Time = 325.421 seconds.

> perf2.data:
> 8. started recording again (no reboot since the previous session,
> so still stuck at 800 MHz)

Time = 393 seconds.

All CPUs are in the trace. What happened in between?

No, frequency stuck at 600 to 700 MHz.

> 9. put laptop on AC

It seems to go into performance mode at some point. The CPU frequency goes up, but jumps around, however 2.6 GHz is typical.

> 10. suspended
> 11. resumed, frequencies went back to 3.2. GHz

First it (CPU 0) goes to 2.4GHz. Again, I think the driver never sends the target pstate because it thinks it didn't change.
This time it kicks out (ref: 408.5609 sec), but I do not know why.
Normal operation resumes (for CPU 0), however I think that the processor think all cores are active, and so the limit is 3.2GHz.

> 12. recording finished
> I hope it is useful.

Yes, that was extremely useful.

(In reply to Piotr Kołaczkowski from comment #74)
> Kernel 4.1-rc4 did not help.

Hmmm... There was one mistake discovered after RC4 was released.
It doesn't effect me, but I only use my proposed patch set now.
It is late in my time zone, I'll get you to try my patch set tomorrow.

> Today I booted laptop on battery once again on kernel 4.1.0-rc4, then
> suspended and resumed and found it locked at 800 MHz. Interestingly I
> observed that immediately after resume, max frequency was 1.8 GHz and it
> changed to 800 MHz a few seconds later, so I guess it was triggered by
> switching the governor to powersave by the script.

Yes, exactly.
Are all CPUs on line after suspend now? You can tell by trying to assign any task to a specific CPU with taskset. It will complain if the CPU is off-line.

> All CPUs are in the trace. What happened in between?

I have no idea, I didn't do anything special like suspend/hibernate/reboot.
I'll repeat the experiments with tracing 4.1-rc4, but I need to compile your tracing patch in first. 

> No, frequency stuck at 600 to 700 MHz.

So maybe i7z reports them wrong, because it showed all multipliers 8x exactly, and reported frequency 750-800 MHz. For sure, in that state it never exceeded 800 MHz by a single Hz. Next time, besides the perf record, I'll record turbostat output.

(In reply to Piotr Kołaczkowski from comment #77)
> So maybe i7z reports them wrong,

Yes, I think so. I have seen i7z report incorrect frequencies before.
That being said, it is hard to know what information to trust when things are in such a messed up state. TSC seems accurate, and MPERF seems to just track TSC (i.e. CPU seems to be locked in C0 state, regardless of load), but APERF seems lower than the minimum 8X would allow.

I had to rebase my proposed patch set for Kernel 4.1. The rebased version can be found at:

double u double u double u dot smythies dot com /~doug/linux/intel_pstate/rebase_to_k4.1/

You are already using patch 1 of 5.

First observations from 4.1-rc4 with all your patches 1-5.

1. scaling_governor is fixed on performance and doesn't automatically drop to powersave anymore
2. turboboost was working when booted freshly on AC
3. lost turboboost on the first suspend on battery
4. turboboost doesn't come back even after putting on AC and suspending

So far no severe lockup (only missing turboboost), but too early to say for sure. I need to do more experiments.

intel_pstate/no_turbo = 0
intel_pstate/max_perf_pct = 100
intel_pstate/min_perf_pct = 100
Frequency locked at 2400 MHz

m4600 intel_pstate # turbostat -d sleep 10
turbostat version 4.1 10-Feb, 2015 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 13 CPUID levels; family:model:stepping 0x6:2a:7 (6:42:7)
CPUID(6): APERF, DTS, PTM, EPB
RAPL: 1456 sec. Joule Counter Range, at 45 Watts
cpu0: MSR_NHM_PLATFORM_INFO: 0x80060011800
8 * 100 = 800 MHz max efficiency
24 * 100 = 2400 MHz TSC frequency
cpu0: MSR_IA32_POWER_CTL: 0x0004005f (C1E auto-promotion: ENabled)
cpu0: MSR_NHM_SNB_PKG_CST_CFG_CTL: 0x1e008404 (UNdemote-C3, UNdemote-C1, demote-C3, demote-C1, locked: pkg-cstate-limit=4: pc7)
cpu0: MSR_NHM_TURBO_RATIO_LIMIT: 0x20202223
32 * 100 = 3200 MHz max turbo 4 active cores
32 * 100 = 3200 MHz max turbo 3 active cores
34 * 100 = 3400 MHz max turbo 2 active cores
35 * 100 = 3500 MHz max turbo 1 active cores
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000000 (performance)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a1003 (0.125000 Watts, 0.000015 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x10024001200168 (45 W TDP, RAPL 36 - 72 W, 0.015625 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x800081c200dc8168 (locked)
cpu0: PKG Limit #1: ENabled (45.000000 Watts, 28.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: ENabled (56.250000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 0
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 0
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x00640e00 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88270000 (61 C)
cpu0: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1)
cpu1: MSR_IA32_THERM_STATUS: 0x88280000 (60 C +/- 1)
cpu2: MSR_IA32_THERM_STATUS: 0x88270000 (61 C +/- 1)
cpu3: MSR_IA32_THERM_STATUS: 0x882c0000 (56 C +/- 1)
    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt
       -       -    2042   85.24    2396    2396       0   13.58    0.32    0.02    0.84      65      65    0.00    0.00    0.00    0.00   28.93   24.16    0.09
       0       0    2211   92.26    2397    2397       1    7.48    0.07    0.01    0.18      62      65    0.00    0.00    0.00    0.00   28.93   24.16    0.09
       0       4    1977   82.54    2396    2396       1   17.19
       1       1    2007   83.77    2395    2395       1   13.14    0.77    0.05    2.26      63
       1       5    1901   79.36    2395    2395       1   17.55
       2       2    2018   84.22    2396    2396       1   15.39    0.17    0.00    0.21      65
       2       6    2256   94.18    2396    2396       1    5.44
       3       3    1873   78.18    2396    2396       1   20.85    0.25    0.00    0.71      59
       3       7    2094   87.41    2396    2396       1   11.62
10.005260 sec

(In reply to Piotr Kołaczkowski from comment #79)
> First observations from 4.1-rc4 with all your patches 1-5.
> 
> 1. scaling_governor is fixed on performance and doesn't automatically drop
> to powersave anymore

That is not related to my patch set. And if you are in performance mose, then my patch set is not being used.

> 2. turboboost was working when booted freshly on AC
> 3. lost turboboost on the first suspend on battery
> 4. turboboost doesn't come back even after putting on AC and suspending

The initialization mess (or lack thereof) upon exit from suspend is not addressed with my patch set.
It will only prevent the one math lock up condition your processor is prone to.

By the way, I forgot to mention, I use a different set of default parameter settings now, although it has yet to be decided what we do if the patch set does end up accepted. I'll put a script that sets them all into that web location where I put the patch set.

Ah, ok, that makes sense. Then I need to check why it doesn't do into powersave mode now.

(In reply to Doug Smythies from comment #75)
> Executive Summary:
> . There are multiple problems.
...
> . There might be problems (not sure on this one) where the intel_pstate
> driver thinks the new target pstate is the same as previously sent, so it
> doesn't send the new one.
...

This scenario has now been verified.

If one just looks at the code, it looks as though every condition is covered.
However, if I just comment out the code in "intel_pstate_set_pstate". i.e.

/*
        if (pstate == cpu->pstate.current_pstate)
                return;
*/

Then things work fine upon exit from suspend (using the performance mode method of messing thing up (the "Marien" method from another bug report)).

Now I am trying to figure out how it gets into this state: Is it some loophole in the intel_pstate driver? Does it try to set the value too early in the process, and the processor clobbers it itself later? Does some other code or process clobber the value later?

While things appear as though turbo is disabled, I actually don't think that is the case, I think it is just that all target pstates somehow get set to the non-turbo maximum.

(In reply to Doug Smythies from comment #83)
> (In reply to Doug Smythies from comment #75)
> > Executive Summary:
> > . There are multiple problems.
> ...
> > . There might be problems (not sure on this one) where the intel_pstate
> > driver thinks the new target pstate is the same as previously sent, so it
> > doesn't send the new one.
> ...
> 
> This scenario has now been verified.

Actually, the logic error is rather glaring, don't know why I didn't see it earlier.

I believe the proper way to deal with this issue is with a patch that I just added to another bug report (attachment 177601 [details] of bug 90421).

@Piotr: There are still some things in your trace data, and your comments, that are cause for concern. Could you add the patch referenced in my previous comment (call it patch 6 of 5), and acquire some new trace data.

Yes, I will over the weekend. Currently running kernel 4.1-rc4 with your patches 1-5, and it feels like performance on powersave is lower, but I haven't any hard data to back it up yet. I had a busy week and didn't have much time to dig into that.

Piotr and I have been doing a lot of work in the background on this.
The great thing is that Piotr has an absolutely repeatable method for creating the low frequency issue.
 
O.K. here is a status update:

1.) With a reasonable degree of confidence, I claim that the low frequency issue is a Dell BIOS problem.

With the current intel_pstate driver, sometimes the CPU frequency locks at less than what should be the minimum for the processor, typically by a factor of between about 0.75 to 0.8.

With the intel_pstate driver with my patch set, the frequency does not lock, but is typically (not always) about 0.75 to 0.8 times what is asked for.

The earliest posting about this issue that I could find was from late 2012, which pre-dates the existence of the intel_pstate driver.

The issue can, and does, come up in mysterious ways. I have trace data from Piotr, which was supposed to be a control sample where everything was fine, but in fact the issue comes and goes a few times.

2.) Issue 1 aside, and as mentioned in my comment 84 above, the intel_pstate driver can lock in such a way that it is out of synchronization with the actual processor target pstate register. The fix patch has been tested and submitted.

I own a Dell E7440, which has an Intel i7-4600U CPU (3.3GHz turbo), and I experience this problem regularly, where the CPU will be locked at very low performance level after resume from suspend (500–600 MHz). It happens seemingly at random and the only way I have found to fix it is to do another suspend, and it will usually be OK on resume again. The kernel uses the 'intel_pstate' driver with 'powersave' governor.

Interestingly, Dell recently released a BIOS update for my laptop which claims to address the issue:
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverID=RYJTF&productCode=latitude-e7440-ultrabook

Which mentions:
> -Fixed CPU may stay at low speed after resume from standby.

Unfortunately, the BIOS update did not correct the issue. I am on Ubuntu 14.04 and the 3.16.X-series default kernel.

I've just experienced the same issue with the replacement laptop, which is equipped with an Ivy Bridge i7-3520M. After suspend-to-ram on battery it's frequency is stuck at 900Mhz. The system is running fedora-21 on linux 4.0.4.


It is a Dell E6330 (more or less the direct successor to the Sandy Bridge E6320 which led to the creation of the bug-report in the first place).

Created attachment 180021 [details]
attachment-28197-0.html

Did you test with Doug's patches or without? They fix the problem of
getting stuck at a very low frequency for me (not perfectly though, but we
think the remaining problem is bios fault). You need to compile kernel from
source.
15 cze 2015 20:49 <bugzilla-daemon@bugzilla.kernel.org> napisał(a):

> https://bugzilla.kernel.org/show_bug.cgi?id=62851
>
> --- Comment #89 from Clemens Eisserer <linuxhippy@gmail.com> ---
> I've just experienced the same issue with the replacement laptop, which is
> equipped with an Ivy Bridge i7-3520M. After suspend-to-ram on battery it's
> frequency is stuck at 900Mhz. The system is running fedora-21 on linux
> 4.0.4.
>
>
> It is a Dell E6330 (more or less the direct successor to the Sandy Bridge
> E6320
> which led to the creation of the bug-report in the first place).
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

I haven't tested the patches yet, however I reported the issue because this was on an Ivy Bridge based notebook, whereas as far as I have seen, all previous issues were experienced on Sandy Bridge.

@oyvinst: Thanks for your posting. Please also see (and maybe add to):
http://en.community.dell.com/support-forums/laptop/f/3518/t/19634759

@Clemens: Thanks for your update postings. What I find intriguing is the 900MHz, which seems to be some new (at least as far as I know) lock up CPU frequency.

Could the people still experiencing low CPU frequencies after suspend please do the following and report back:

1.) (needed once per boot) sudo modprobe msr
2.) before any suspend: sudo rdmsr -a 0x19a
3.) after a suspend that results in the low CPU frequencies:
 sudo rdmsr -a 0x19a
4.) If the result from step 3 is not 0, then:
 sudo wrmsr -a 0x19a 0x0
and check it:
 sudo rdmsr -a 0x19a
5.) Are the CPU frequencies O.K. now?

For people still experiencing low CPU frequencies after suspend:

In addition to my request in comment 93, the Intel subject matter expert on thermal interactions and pstates also wants the output from:

cd /sys/class/thermal
grep -r . *

Just experienced this issue on a Dell Latitude E7240 (Haswell i7-4600U) running linux-4.1.7, cores are stuck between 500-600mhz.

The output of /sys/class/thermal is:

cooling_device0/type:Processor
cooling_device0/power/control:auto
cooling_device0/power/async:disabled
cooling_device0/power/runtime_enabled:disabled
cooling_device0/power/runtime_active_kids:0
cooling_device0/power/runtime_active_time:0
grep: cooling_device0/power/autosuspend_delay_ms: Input/output error
cooling_device0/power/runtime_status:unsupported
cooling_device0/power/runtime_usage:0
cooling_device0/power/runtime_suspended_time:0
cooling_device0/cur_state:0
cooling_device0/max_state:3
cooling_device1/type:Processor
cooling_device1/power/control:auto
cooling_device1/power/async:disabled
cooling_device1/power/runtime_enabled:disabled
cooling_device1/power/runtime_active_kids:0
cooling_device1/power/runtime_active_time:0
grep: cooling_device1/power/autosuspend_delay_ms: Input/output error
cooling_device1/power/runtime_status:unsupported
cooling_device1/power/runtime_usage:0
cooling_device1/power/runtime_suspended_time:0
cooling_device1/cur_state:0
cooling_device1/max_state:3
cooling_device2/type:Processor
cooling_device2/power/control:auto
cooling_device2/power/async:disabled
cooling_device2/power/runtime_enabled:disabled
cooling_device2/power/runtime_active_kids:0
cooling_device2/power/runtime_active_time:0
grep: cooling_device2/power/autosuspend_delay_ms: Input/output error
cooling_device2/power/runtime_status:unsupported
cooling_device2/power/runtime_usage:0
cooling_device2/power/runtime_suspended_time:0
cooling_device2/cur_state:0
cooling_device2/max_state:3
cooling_device3/type:Processor
cooling_device3/power/control:auto
cooling_device3/power/async:disabled
cooling_device3/power/runtime_enabled:disabled
cooling_device3/power/runtime_active_kids:0
cooling_device3/power/runtime_active_time:0
grep: cooling_device3/power/autosuspend_delay_ms: Input/output error
cooling_device3/power/runtime_status:unsupported
cooling_device3/power/runtime_usage:0
cooling_device3/power/runtime_suspended_time:0
cooling_device3/cur_state:0
cooling_device3/max_state:3
thermal_zone0/mode:enabled
thermal_zone0/temp:25000
thermal_zone0/type:acpitz
thermal_zone0/power/control:auto
thermal_zone0/power/async:disabled
thermal_zone0/power/runtime_enabled:disabled
thermal_zone0/power/runtime_active_kids:0
thermal_zone0/power/runtime_active_time:0
grep: thermal_zone0/power/autosuspend_delay_ms: Input/output error
thermal_zone0/power/runtime_status:unsupported
thermal_zone0/power/runtime_usage:0
thermal_zone0/power/runtime_suspended_time:0
thermal_zone0/trip_point_0_temp:107000
thermal_zone0/trip_point_0_type:critical
thermal_zone0/policy:step_wise
thermal_zone0/passive:0
thermal_zone1/temp:45000
thermal_zone1/type:x86_pkg_temp
thermal_zone1/power/control:auto
thermal_zone1/power/async:disabled
thermal_zone1/power/runtime_enabled:disabled
thermal_zone1/power/runtime_active_kids:0
thermal_zone1/power/runtime_active_time:0
grep: thermal_zone1/power/autosuspend_delay_ms: Input/output error
thermal_zone1/power/runtime_status:unsupported
thermal_zone1/power/runtime_usage:0
thermal_zone1/power/runtime_suspended_time:0
thermal_zone1/trip_point_0_temp:0
thermal_zone1/trip_point_0_type:passive
thermal_zone1/trip_point_1_temp:0
thermal_zone1/trip_point_1_type:passive
thermal_zone1/policy:step_wise

@Clemens: Please do my comment number 93 stuff.

Were you on battery? And was this after a resume from suspend?

Note: there is a mistake in step 4. It should be "If the result from step 3 has bit 4 set, then:
 sudo wrmsr -a 0x19a 0x0

There have been cases where the rdmsr returns non-zero values, where Clock Modulation (bit 4) is not enabled.

(In reply to Doug Smythies from comment #93)
> Could the people still experiencing low CPU frequencies after suspend please
> do the following and report back:
> 
> 1.) (needed once per boot) sudo modprobe msr
> 2.) before any suspend: sudo rdmsr -a 0x19a
> 3.) after a suspend that results in the low CPU frequencies:
>  sudo rdmsr -a 0x19a
> 4.) If the result from step 3 is not 0, then:
>  sudo wrmsr -a 0x19a 0x0
> and check it:
>  sudo rdmsr -a 0x19a
> 5.) Are the CPU frequencies O.K. now?

ping clemems.

I haven't experienced this issue for quite a while running fedora stock kernels.
As soon as I hit it again, I'll try to perform the steps mentioned in comment 93.

Hmm, okay. I will close this bug for now.
Please feel free to re-open it if you can reproduce the problem in vanilla kernel.