Dual-Xeon Nehalem EP system from supermicro, sitting idle. BIOS version 10/06/2009 In this example, booted with "maxcpus=1" Powertop says it is almost all in C3 mwait, and at 1.60 GHz: Cn Avg residency C0 (cpu running) ( 0.0%) polling 76.9ms ( 6.7%) C1 halt 0.2ms ( 0.0%) C2 0.0ms ( 0.0%) C3 mwait 70.3ms (93.3%) P-states (frequencies) 2.80 Ghz 0.3% 2.80 Ghz 0.0% 2.67 Ghz 0.0% 2.53 Ghz 0.0% 1.60 Ghz 99.7% However, turbostat http://ftp.kernel.org/pub/linux/kernel/people/lenb/acpi/utils/pmtools-latest/turbostat/ shows the system is running at 2.8 GHz and never goes deeper than c1: # ./turbostat -v CPUID GenuineIntel 11 levels family:model:stepping 6:26:5 Nehalem multiplier 21, TSC frequency 2800 MHz Nehalem 4 cores active: 23 mult, max turbo frequency = 3067 MHz Nehalem 3 cores active: 23 mult, max turbo frequency = 3067 MHz Nehalem 2 cores active: 24 mult, max turbo frequency = 3200 MHz Nehalem 1 core active: 24 mult, max turbo frequency = 3200 MHz CPU GHz TSC %c0 %c1 %c3 %c6 %pc3 %pc6 %pc7 0 2.80 2.80 0.09 99.91 0.00 0.00 0.00 0.00 0.00 Further, when a cycle soaker is executed: # cat /dev/zero > /dev/null the turbo frequency tops out at 3.07 GHz, when we would hope for 3.2 Ghz: # ./turbostat CPU GHz TSC %c0 %c1 %c3 %c6 %pc3 %pc6 %pc7 0 3.07 2.80 100.00 0.00 0.00 0.00 0.00 0.00 0.00
booting without maxcpus=1, this problem goes away. All the cores are able to enter c6 and get their frequency down to 1.6GHz for the short time they are in c0: CPU GHz TSC %c0 %c1 %c3 %c6 %pc3 %pc6 %pc7 0 1.60 2.80 0.16 0.23 0.00 99.61 0.00 98.95 0.00 1 1.60 2.80 0.08 0.09 0.00 99.83 0.00 98.95 0.00 2 1.60 2.80 0.01 0.02 0.00 99.96 0.00 98.95 0.00 3 1.63 2.80 0.01 0.03 0.00 99.96 0.00 98.95 0.00 4 1.60 2.80 0.01 0.02 0.00 99.98 0.00 98.96 0.00 5 1.60 2.80 0.06 0.13 0.00 99.81 0.00 98.96 0.00 6 1.58 2.80 0.01 0.02 0.00 99.97 0.00 98.96 0.00 7 1.60 2.80 0.01 0.02 0.00 99.98 0.00 98.96 0.00 8 1.60 2.80 0.04 0.35 0.00 99.61 0.00 98.95 0.00 9 1.60 2.80 0.02 0.15 0.00 99.83 0.00 98.95 0.00 10 1.60 2.80 0.01 0.03 0.00 99.96 0.00 98.95 0.00 11 1.59 2.80 0.01 0.03 0.00 99.96 0.00 98.95 0.00 12 1.57 2.80 0.01 0.02 0.00 99.98 0.00 98.96 0.00 13 1.60 2.80 0.04 0.15 0.00 99.81 0.00 98.96 0.00 14 1.63 2.80 0.01 0.02 0.00 99.97 0.00 98.96 0.00 15 1.60 2.80 0.01 0.01 0.00 99.98 0.00 98.96 0.00 When a cycle soaker is run, it is able to turbo-up to 3.2GHz as expected: CPU GHz TSC %c0 %c1 %c3 %c6 %pc3 %pc6 %pc7 0 3.20 2.80 1.91 6.24 3.63 88.22 0.00 0.00 0.00 1 3.19 2.80 0.05 99.95 0.00 0.00 0.00 0.00 0.00 2 2.90 2.80 0.02 0.09 84.97 14.92 0.00 0.00 0.00 3 2.49 2.80 0.01 0.09 84.98 14.92 0.00 0.00 0.00 4 1.60 2.80 0.14 0.21 0.00 99.65 0.00 0.00 0.00 5 1.60 2.80 0.05 0.10 0.00 99.85 0.00 0.00 0.00 6 1.63 2.80 0.01 0.02 0.00 99.98 0.00 0.00 0.00 7 1.60 2.80 0.01 0.02 0.00 99.98 0.00 0.00 0.00 8 3.20 2.80 2.24 5.91 3.63 88.22 0.00 0.00 0.00 9 3.20 2.80 99.99 0.01 0.00 0.00 0.00 0.00 0.00 10 2.53 2.80 0.01 0.10 84.98 14.91 0.00 0.00 0.00 11 2.51 2.80 0.01 0.09 84.98 14.92 0.00 0.00 0.00 12 1.60 2.80 0.07 0.29 0.00 99.65 0.00 0.00 0.00 13 1.60 2.80 0.04 0.10 0.00 99.85 0.00 0.00 0.00 14 1.60 2.80 0.01 0.02 0.00 99.98 0.00 0.00 0.00 15 1.60 2.80 0.01 0.01 0.00 99.98 0.00 0.00 0.00 The "maxcpus=1" power-management failure is due to a BIOS bug. The BIOS is supposed to put non-boot CPUSs into c6 so that if the OS does not start them, they will not interfere with power management. However, this BIOS is clearly leaving the non-boot CPUs in c1, which interferes with the power-management of the online cpus. This same symptom is seen when a kernel built for MAX_CPUS=8 is run on a 16-thread system (which is how I first noticed this problem).