Bug 8842 - thermal throttling at 50C impacts performance - AOpen i915GMm-hfs, Pentium-M 750
Summary: thermal throttling at 50C impacts performance - AOpen i915GMm-hfs, Pentium-M 750
Status: CLOSED CODE_FIX
Alias: None
Product: ACPI
Classification: Unclassified
Component: BIOS (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Len Brown
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-08-04 00:09 UTC by Knut Petersen
Modified: 2007-08-13 21:42 UTC (History)
3 users (show)

See Also:
Kernel Version: 2.6.22
Tree: Mainline
Regression: Yes


Attachments
acpidump file (83.06 KB, application/octet-stream)
2007-08-04 00:14 UTC, Knut Petersen
Details
dmidecode output (11.73 KB, application/octet-stream)
2007-08-04 00:14 UTC, Knut Petersen
Details
disassembled dsdt (164.00 KB, text/x-dsl)
2007-08-04 00:15 UTC, Knut Petersen
Details
patch vs 2.6.23-rc2 to create thermal.psv (3.06 KB, patch)
2007-08-06 23:02 UTC, Len Brown
Details | Diff
patch vs 2.6.23-rc2 to create thermal.disable=1 bootparam (3.00 KB, patch)
2007-08-07 15:39 UTC, Len Brown
Details | Diff
patch vs 2.6.23-rc2 (6.88 KB, patch)
2007-08-08 22:48 UTC, Len Brown
Details | Diff
patch vs 2.6.23-rc2 (6.80 KB, text/x-diff)
2007-08-09 10:33 UTC, Len Brown
Details

Description Knut Petersen 2007-08-04 00:09:26 UTC
Most recent kernel where this bug did not occur: 2.6.21

Distribution: openSuSE 10.2

Hardware Environment: AOpen i915GMm-hfs mainboard, Pentium-M 750. Although
a Pentium M and a mobile chipset is used, don´t believe that this is a
laptop. It´s a normal ATX board for server/desktop/htpc use!

Problem Description: The mainboard BIOS is broken, but that was not
a problem as long as it was possible to provide thermal.c with some
more reasonable data by writing to the trip_points file. Now there 
is a 50% performance regression - kernel compile time increase:
2.6.21: 12min, 2.6.22.1 18 min.

mainboard details
------------------
thermal.c is unable to control the cpu fan and complains about that:

ACPI: Unable to turn cooling device [c18dc9dc] 'on'
bus-0201 [-411] bus_set_power: Device `[PNP0C0B:00]' is not power manageable

The BIOS provides the following trip points for both passive and active
cooling mode:

critical (S5):           100 C
passive:                 50 C: tc1=4 tc2=3 tsp=60 devices=CPU0
active[0]:               50 C: devices= FAN

These trip points are stored as dK values in the BIOS data:
DSDT: OperationRegion (TEMM, SystemMemory, 0x000FF810, 0x0C)

linux:~ # hexdump -C -s 0x000FF810  -n 12 /dev/mem
000ff810  a0 0c 6e 0c a0 0c 6e 0c  94 0e 01 00              |..n...n.....|

The BIOS will turn on the cpu fan a bit above 50 degrees Celsius, but
the passive cooling of thermal.c slows down the cpu before that occures.

Linux problems
--------------
The cause is commit 11ccc0f249cb01a129f54760b8ff087f242935d4,
removing te possibility to write to the trip_points file.

In kernels up to 2.6.21 it was possible to use the following trip point setup
critical (S5):           100 C
passive:                 60 C: tc1=4 tc2=3 tsp=60 devices=CPU0
active[0]:               70 C: devices= FAN

The result was that the BIOS would switch on the fan early enough that
neither passive nor active cooling of thermal.c would be activated as long
as the fan was working properly. But in case of fan failure my system would
have been still in a slower but safe state at about 60 degrees Celsius,
ensured by passive cooling code in thermal.c. That is not possible any
longer with kernel 2.6.22. Maybe the BIOS would save my system at
100 degrees Celsius, but I definitely will not test that.
Comment 1 Knut Petersen 2007-08-04 00:14:04 UTC
Created attachment 12246 [details]
acpidump file
Comment 2 Knut Petersen 2007-08-04 00:14:47 UTC
Created attachment 12247 [details]
dmidecode output
Comment 3 Knut Petersen 2007-08-04 00:15:47 UTC
Created attachment 12248 [details]
disassembled dsdt
Comment 4 Knut Petersen 2007-08-04 00:32:12 UTC
No, I do not want to patch the binary BIOS with reasonable trip points.
The flash ROM is soldered to the board, one mistake would kill the system.

There are more people who need to override BIOS supplied trip points,
see lkml.

Please don´t tell me that overriding the dsdt with a version that was
edited by Dr. Fred Mbogo is safer than the old writable trip_points file.
Comment 5 Andrey Melentyev 2007-08-06 15:20:09 UTC
I don't know if my trouble is in the same thing as Knut's, but I have the following situation:

from /proc/acpi/thermal_zone/THRM/:
# cat *
0 - Active; 1 - Passive
<polling disabled>
state:                   ok
temperature:             45 C
critical (S5):           110 C
passive:                 78 C: tc1=3 tc2=1 tsp=80 devices=CPU0
active[0]:               60 C: devices= FAN

Even when laptop's CPU is 100% busy for a long time, I can't get a temperature more than 55 C. The problem is that CPU fan is always running at full speed. It's very noise. I can't control it via /proc/acpi/fan/FAN/state and /proc/acpi/thermal_zone/THRM/cooling_mode too.
My laptop is based on some FIC model and I've got the latest BIOS version from the manufacturer site.

Under Windows the behavior of the CPU fan differs from this. The fan is quiet almost all the time.

If it is the same bug, I can attach my acpidump and dmidecode output here. If it is not, I apologise for the interference and I'll post another one.
Comment 6 Len Brown 2007-08-06 20:22:08 UTC
> ACPI: Unable to turn cooling device [c18dc9dc] 'on'
> bus-0201 [-411] bus_set_power: Device `[PNP0C0B:00]' is not power manageable

Knut,
Are you running with (latest) BIOS SETUP options all in default settings?
Are there any fan-related options such as quiet vs performance settings?

Please attach the complete output from dmesg -s64000
including the cooling device message above.
(the bus_set_power message is unrelated)

What do you see in /proc/acpi/fan/*/state before and after the
temperature crosses the 50C threshold -- is that when
the message above comes out, or does that occur upon boot?
Does the fan actually come on automatically when you reach 50,
or at another temperature?
If you echo 0 and 3 into the the fan state files(s) are you able
to have any effect on the fan state?

Curiously, the DSDT has an SFAN method with FON/FOFF
which are likely for controlling the fan, but these
methods are not referenced.

Looking over the DSDT, it does appear that this system intends
to support passive and active cooling modes.   However, both
the trip_points file and the dd above agree that the BIOS
is for some reason initializing both the active and passive
high-trip points to 50C.  Curiously, the DSDT records low
trip points that dd shows are 46C, but it doesn't seem to
implement hysteresis -- at least not in AML.
It would be interesting to observe at what temperature the
fan turns off at when the system cools.
It would be interesting if any system events at all had
any effect on the output of the dd above.

Please echo 1 and 0 into /proc/acpi/thermal_zone/*/cooling_mode
While the AML will not change the trip-points, it is possible
that something needs to trigger an SMI that would change
them and then updated trip-points would be picked up by AML
from the region you describe above. (and thus they'd be
visible in the trip_points file)

BTW. there is no danger of harming the flash soldered onto the board
by using dd to over-write the underlying trip-points above.
If that region is mapped to FLASH, then dd would simply fail.
If it works, then you'd be updating writable memory
(and the effect would go away after reboot/reset)
However, you'd have to notify AML of the update, such
as by executing an _SCP by writing to the cooling_mode file
as above.  Further, it is possible that SMM may overwrite
the value that you write...

The other question is where and what thermal events this
machine actually produces and what controls them.
Please kill acpid and cat /proc/acpi/event and see
what thermal events you get as the temperature changes.
(you can press the power button to make sure this scheme
 is working -- /proc/interrupts should show the acpi line
 increment for each press, and you should get a text line
 out of /proc/acpi/event)

When you boot with "acpi=off", does the system boot properly,
run full load at full speed, and enable the fan as needed?
Can you detect from the sound if the fan speed in this mode
is the about same as the fan speed in ACPI mode, or can
you perceive a speed difference?

Finally, please confirm that your kernel has CONFIG_HWMON=n

Andrey,
your issue (fan always on) is different (and less severe) than Knut's.
Please open a new bug report, please attach the output from acpidump.
Comment 7 Len Brown 2007-08-06 21:04:25 UTC
> acpi=off

note that in the event that "acpi=off" causes some unrelated
legacy-mode issues, you can also build a kernel with
CONFIG_ACPI_THERMAL=n to include all of ACPI but exclude the thermal support.
Comment 8 Len Brown 2007-08-06 23:02:20 UTC
Created attachment 12287 [details]
patch vs 2.6.23-rc2 to create thermal.psv

This patch disables ACPI processor thermal throttling on the AOpen system
on the assumption that it is a BIOS bug that _PSV is set down at 50C.

Please verify that it correctly recognizes the system.
If it does not, please invoke it by booting with "thermal.psv=0"

Please verify that the passive trip-point is missing from
/proc/acpi/thermal_zone/*/trip_points, and that that the system
performs at full speed even when it is running above 50C.

Please verify that the fan begins running when the system
heats up above 50C, and that it stops running when the system
cools below 50C.

Please see how high you can push the temperature, given that
the fan should be spinning when you cross 50 C.  Note that
laptops typically invoke ACPI processor thermal throttling
in the neighborhood of 90C.  Note also that Intel Thermal Monitor
will throttle the processor automatically under hardware control
starting at some temperature below 100C.

The main utility of ACPI processor thermal throttling is
to allow for a passively cooled system where the user is
willing to sacrafice performance to reduce fan noise.

Note that if the DMI entry is correct, you can disable the effect
of this patch by by booting with "thermal.psv=1"
Comment 9 Knut Petersen 2007-08-07 00:15:53 UTC
    >Are you running with (latest) BIOS SETUP options all in default settings?
    >Are there any fan-related options such as quiet vs performance settings?

    In the BIOS setup not everything is default. It is possible to select between
    three modes:
    - all fans at a configurable speed
    - all fans at full speed
    - smart mode (fans are switched on and off dependent on temperature.

    In all three modes the trip points reported to linux are as described earlier.

    >Please attach the complete output from dmesg -s64000
    >including the cooling device message above.
    >(the bus_set_power message is unrelated)

    Well, it is not unrelated. thermal.c is trying to activate the fan, that
    fails and triggers the message:

    Jul 31 10:29:40 linux kernel: [  857.048000]  thermal-0738 [-410] thermal_check
            : THRM: temperature[3222] sleep[2000]
    [  859.048000] Execute Method: [\_TZ_.THRM._TMP] (Node c18dc8ec)
    [  859.048000] exregion-0185 [-396] ex_system_memory_space: System-Memory
    (width 16) R/W 0 Address=C025D8350$
    [  859.048000] exregion-0287 [-395] ex_system_io_space_han: System-IO (width 8)
    R/W 1 Address=C026644B000002$
    [  859.048000] exregion-0287 [-395] ex_system_io_space_han: System-IO (width 8)
    R/W 1 Address=C026644B000002$
    [  859.048000] exregion-0287 [-395] ex_system_io_space_han: System-IO (width 8)
    R/W 1 Address=C026644B000002$
    [  859.048000] exregion-0287 [-396] ex_system_io_space_han: System-IO (width 8)
    R/W 0 Address=C026644B000002$
    [  859.048000] exregion-0185 [-396] ex_system_memory_space: System-Memory
    (width 8) R/W 0 Address=C025D83500$
    [  859.048000]    utils-0287 [-411] evaluate_integer      : Return value [3242]
    [  859.048000]  thermal-0230 [-411] thermal_get_temperatur: Temperature is 3242
    dK
    [  859.048000]  thermal-0511 [-411] thermal_passive       :
    trend[110]=(tc1[4]*(tmp[3242]-last[3222]))+(tc2[$
    [  859.048000]      bus-0201 [-411] bus_set_power         : Device
    `[PNP0C0B:00]' is not power manageable
    [  859.048000] ACPI: Unable to turn cooling device [c18dc9dc] 'on'
    [  859.048000]  thermal-0738 [-411] thermal_check         : THRM:
    temperature[3242] sleep[6000]
    [  865.048000] Execute Method: [\_TZ_.THRM._TMP] (Node c18dc8ec)

    >What do you see in /proc/acpi/fan/*/state before and after the
    >temperature crosses the 50C threshold -- is that when
    > the message above comes out, or does that occur upon boot?

    Fan state is always on, no matter if the fan is running or not. Writing to
    /proc/acpi/fan/FAN/state fails. As you can see from the system log snippet
    above, the message is only triggered if a more verbose acpi debug mode is
    selected and thermal.c tries to activate the fan.


    >Does the fan actually come on automatically when you reach 50,
    >or at another temperature?

    In "smart" mode, the fan comes on at about the selected temperature. But it´s
    not a fast response. Otherwise the cpu fan runs always at full speed or at the
    BIOS selected speed.

    >If you echo 0 and 3 into the the fan state files(s) are you able
    >to have any effect on the fan state?

    linux:/var/log # echo 3 > /proc/acpi/fan/FAN/state
    -bash: echo: write error: No such device

    No effect.

    >Curiously, the DSDT has an SFAN method with FON/FOFF
    >which are likely for controlling the fan, but these
    >methods are not referenced.

    Yes, isn´t that nice? ;-))

    >Looking over the DSDT, it does appear that this system intends
    >to support passive and active cooling modes.   However, both
    >the trip_points file and the dd above agree that the BIOS
    >is for some reason initializing both the active and passive
    >high-trip points to 50C.  Curiously, the DSDT records low
    >trip points that dd shows are 46C, but it doesn't seem to
    >implement hysteresis -- at least not in AML.
    >It would be interesting to observe at what temperature the
    >fan turns off at when the system cools.

    Fan on: For some seconds a bit above 50 deg. Celsius
    Fan off: Fore some secondes a bit below that temperature
    Switch off temp somewhere in the range 45 ... 47.

    >It would be interesting if any system events at all had
    >any effect on the output of the dd above.

    >Please echo 1 and 0 into /proc/acpi/thermal_zone/*/cooling_mode
    >While the AML will not change the trip-points, it is possible
    >that something needs to trigger an SMI that would change
    >them and then updated trip-points would be picked up by AML
    >from the region you describe above. (and thus they'd be
    >visible in the trip_points file)

    Echoing 0 and 1 to cooling_mode does reset the trip points to those
    stored in the BIOS data (50/50 deg. Celsius). That´s the behaviour
    programmed in the DSDT, isn´t it?

    >BTW. there is no danger of harming the flash soldered onto the board
    >by using dd to over-write the underlying trip-points above.
    >If that region is mapped to FLASH, then dd would simply fail.
    >If it works, then you'd be updating writable memory
    >(and the effect would go away after reboot/reset)
    >However, you'd have to notify AML of the update, such
    >as by executing an _SCP by writing to the cooling_mode file
    >as above.  Further, it is possible that SMM may overwrite
    >the value that you write...

    Well, the memory at 0x000FF810 is read only. As I was talking
    about patching the BIOS I meant to unpack an AOpen bios file,
    to patch the main BIOS image, to recalculate the check sums, to
    store the patched main BIOS to the compressed BIOS image and to
    flash that product to the chip. I had to do that for several
    computers years ago to overcome hard disk size limits, but I
    always had a spare programmed flash with an original BIOS.

    >The other question is where and what thermal events this
    >machine actually produces and what controls them.
    >Please kill acpid and cat /proc/acpi/event and see
    >what thermal events you get as the temperature changes.
    >(you can press the power button to make sure this scheme
    > is working -- /proc/interrupts should show the acpi line
    > increment for each press, and you should get a text line
    > out of /proc/acpi/event)

    There are no thermal events as the trip points are crossed,
    but as thermal.c polls every two seconds it detects the 
    temperature changes. Writing to cooling_mode produces an event,
    but not an increment in the irq 9 (acpi) count.

    >When you boot with "acpi=off", does the system boot properly,
    >run full load at full speed, and enable the fan as needed?

    It does boot and work at full speed, but some devices do not work.
    That includes but is not limited to all the USB stuff.

    >Can you detect from the sound if the fan speed in this mode
    >is the about same as the fan speed in ACPI mode, or can
    >you perceive a speed difference?

    The fan is controlled by the BIOS, and the machine is running
    at full speed. 

    >Finally, please confirm that your kernel has CONFIG_HWMON=n

    I tried that, but there is no difference to CONFIG_HWMON=y.
Comment 10 Len Brown 2007-08-07 15:17:52 UTC
> bus-0201 [-411] bus_set_power : Device `[PNP0C0B:00]' is not power manageable
> ACPI: Unable to turn cooling device [c18dc9dc] 'on'

       Device (FAN)
        {
            Name (_HID, EisaId ("PNP0C0B"))
            Method (_INI, 0, NotSerialized)
            {
                Store (TP1H, CTOS)
                Store (TP1L, CTHY)
            }
        }

I've searched my library of DSDT's for fan devices
which are missing _PR0 and found two others that
are broken like this one. (shuttle nforce2, and soyo via)
and both of them are from this BIOS vendor -- Award.

Hmmm, what "Award" should we give this BIOS writer?:-)

This means that the active trip points are worthless
on this machine, as the BIOS is not filling in the hooks
we need to actually control the fan.  Presumably the
fan control you've got is entirely SMM /hardware based.

This means also that the former method of over-riding
the active trip-points on this machine had no effect,
other than giving the operator the impression that they were
in control when they were not...

So how high are you able to drive the temperature
with the patch applied and the passive trip point disabled?
Comment 11 Len Brown 2007-08-07 15:39:53 UTC
Created attachment 12311 [details]
patch vs 2.6.23-rc2 to create thermal.disable=1 bootparam

This patch creates the boot parameter thermal.disable=1,
which is the boot-time equivalent of building with
CONFIG_ACPI_THERMAL=n.

As ACPI thermal control on the AOpen board in this report
is worthless, this parameter is invoked automatically
via DMI for that board.
Comment 12 Len Brown 2007-08-08 10:40:09 UTC
> There are no thermal events as the trip points are crossed

This explains why Windows doesn't fail on this machine.
Windows doesn't poll.  As the machine never sends events,
it doesn't call attention to how broken it is.

Knut,
It occured to me that CONFIG_ACPI_THERMAL=n and thermal.disable=1
may be over-kill, even for a box as broken as this one.
For you may still be interested in reading the current temperature.

Also, if we enable polling, there is no reason that we couldn't keep
the critical trip point.  Indeed, if we have to enable polling,
we could give you a passive trip point override also if you want it.

Active cooling control, of course, is hopeless on this box.
Comment 13 Knut Petersen 2007-08-08 13:27:33 UTC
YES, the attachment to comment #11 is too brutal ;-)

I think the best solution would be to identify some
broken machines and to provide better trip points for
them if needed. Polling must be forced for some of
them, active cooling must be ignored for some. Never
disable passive cooling, never disable critical trip
point handling.

Wouldn´t it be a good idea to test cooling devices
in the init code and to make some sanity checks for
trip points?

More tomorrow.

cu,
 Knut
Comment 14 Len Brown 2007-08-08 22:48:44 UTC
Created attachment 12326 [details]
patch vs 2.6.23-rc2

Please test this patch, it uses DMI to:
1. enable polling (BIOS thermal events are broken)
(note, your distro may write into /proc to do this already,
 but the kernel default is to not enable polling -- except
 for this case)
2. disable active trip points (BIOS fan control is broken)
3. disable passive trip point (BIOS hard-codes it too low)

The actual temperature reading does work,
and with the aid of polling, the critical
trip point should work too.

Note that if you want to specify a passive trip point,
you can override this DMI switch with, say, "thermal.psv=90"
However, I figured selection of an actual value should be
up to the user, and the default would be safer to simply
disable the broken value.

I'll send this patch series to linux-acpi in a moment...
Comment 15 Len Brown 2007-08-08 22:59:38 UTC
BTW. on un-patched 2.6.22, if you
# echo 0 > /proc/acpi/thermal_zone/*/polling_frequency
immediately after boot and before you do the build,
does the performance problem go away?
Comment 16 Len Brown 2007-08-09 10:33:27 UTC
Created attachment 12338 [details]
patch vs 2.6.23-rc2

refresh patch in comment #14

thermal.psv=-1 now disables all passive trip points (was 0)
thermal.psv=degrees-C is now properly converted to internal Kelvin*10 units
Comment 17 Len Brown 2007-08-13 21:42:16 UTC
patch in comment #16 shipped in Linux-2.6.23-rc3
closed.

Note You need to log in before you can comment on or make changes to this bug.