Bug 15559

Summary: Bad performance after suspend on Intel i7 and i5
Product: Platform Specific/Hardware Reporter: Florian Pritz (bluewind)
Component: x86-64Assignee: platform_x86_64 (platform_x86_64)
Status: RESOLVED CODE_FIX    
Severity: normal CC: a.p.zijlstra, alan, chris, debian00, hpa, jonathan.protzenko, mans, psastudio, rjw, stephan.eicher, suresh.b.siddha, sxxe, thomas, venki, wonghow
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.33 Subsystem:
Regression: No Bisected commit-id:
Bug Depends on:    
Bug Blocks: 7216    
Attachments: lspci on the Toshiba laptop
dmidecode on the Toshiba laptop
cpuinfo on the Toshiba laptop
/proc/cpuinfo for i7-920
dmidecode for i7-920
lspci for i7-920
dmesg after suspending 2 times
dmesg after suspending once
lspci laptop HP DV8-1050eg
dmidecode of HP DV8-1050eg
1st Scheduler trace (long)
2nd Scheduler trace (short)
Scheduler trace while building kernel
Simplified workload: a process that spins and reports its speed
add.c (test program)
Patch to root cause the issue
Proposed patch to fix the issue
Second attempt at fixing the issue

Description Florian Pritz 2010-03-17 18:52:55 UTC
Hi,

When I run make -j12 (on a kernel tree) all 8 cores are at 100% each.
(Can also be tested with stress -c 8 or any multithreaded program)
After a suspend2ram (using pm-suspend) only one single core is at 100%
and the others are somewhere between 0% and 10% though not 100% like
before.

This bug is reproduceable on at least 3 machines, running on i7-920,
i7-860 and i5-520M.
Tested kernels are 2.6.33-ARCH (config [1]) , 2.6.32.9-ARCH and
2.6.34-rc1-00005-g522dba7 (only be me; config [2]).

/proc/cpuinfo shows 8 cpus before and after suspend.

My own hardware:
CPU: Intel i7-920
Board: Gigabyte GA-EX58-UD4P [3]
RAM: 8GB
Arch: x86_64 (all 3)
Distribution: Archlinux (all 3)

I don't know of any older good kernel, so sadly can't bisect anything.

Thanks for looking into that issue.

[1]
http://repos.archlinux.org/wsvn/packages/kernel26/trunk/config.x86_64?rev=70484&peg=70484
[2] http://flo.xssn.at/~flo/tmp/kernel-config
[3]
http://www.gigabyte.com.tw/Products/Motherboard/Products_Overview.aspx?ProductID=2986
Comment 1 Rafael J. Wysocki 2010-03-17 20:26:17 UTC
Did you test any kernels older than 2.6.34-rc1 on this box?  If you did, what did they behave the same way?
Comment 2 Florian Pritz 2010-03-17 20:48:25 UTC
I've tested 2.6.33.x 2.6.32.x, but only yet noticed that problem vanished upon reboot. Yes, they all had the same bug.
Comment 3 Thomas Bächler 2010-03-18 18:42:26 UTC
I have only tested 2.6.33 and 2.6.33.1, as this machine is quite new. Besides other unrelated suspend problems, I can always reproduce this one on a Core i5-520M, the symptoms are exactly the ones that Florian describes.

The diversity of hardware and different kernels here suggests that this is a general problem with the i5 and i7 CPUs on Linux. Also interesting is that Florian is using an i7 of the 2009 generation, while I use the 2010 i5 CPU and the bug is the same on both.

On the same kernel(s), my old Core 2 Duo CPU has no trouble after resume, all cores are still being used as usual.

If you need any more info, I think we will be able to provide anything you need. Maybe it might also be useful to bring someone from Intel in on this.
Comment 4 Rafael J. Wysocki 2010-03-18 19:54:28 UTC
Hi,

Here's an interesting issue, apparently Core i5 and i7 have problems after
resume from suspend/standby.

I wonder whom at Intel I should let know about that.

Rafael


On Thursday 18 March 2010, bugzilla-daemon@bugzilla.kernel.org wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=15559
> 
> 
> 
> 
> 
> --- Comment #3 from Thomas Bächler <thomas@archlinux.org>  2010-03-18
> 18:42:26 ---
> I have only tested 2.6.33 and 2.6.33.1, as this machine is quite new. Besides
> other unrelated suspend problems, I can always reproduce this one on a Core
> i5-520M, the symptoms are exactly the ones that Florian describes.
> 
> The diversity of hardware and different kernels here suggests that this is a
> general problem with the i5 and i7 CPUs on Linux. Also interesting is that
> Florian is using an i7 of the 2009 generation, while I use the 2010 i5 CPU
> and
> the bug is the same on both.
> 
> On the same kernel(s), my old Core 2 Duo CPU has no trouble after resume, all
> cores are still being used as usual.
> 
> If you need any more info, I think we will be able to provide anything you
> need. Maybe it might also be useful to bring someone from Intel in on this.
Comment 5 Florian Pritz 2010-03-27 16:44:36 UTC
I've tried disabling HT, but it didn't help. Anyone from Intel yet looked into this issue?
Comment 6 Florian Pritz 2010-04-12 20:55:48 UTC
Hey guys, this bug looks quite critical to me. Looks like all Intel i* systems are affected and the systems are damn slow once it kicks in. 
Could someone if not Intel themselves please take a closer look at the issue?
Comment 7 H. Peter Anvin 2010-04-12 21:03:29 UTC
Please give as many details about the affected systems as possible.  In particular, are they all Gigabyte GA-EX58-UD4P, for example?  What BIOS versions?

This smells like a BIOS problem to me.
Comment 8 Thomas Bächler 2010-04-12 21:19:18 UTC
My system is a Toshiba Tecra A11 Laptop with a Core i5-520M (the new 2010 model), with all BIOS versions I tried so far (I am going to update the BIOS again this week, as Toshiba released a new update last week).

I'll attach lspci, dmidecode, /proc/cpuinfo for this laptop. Anything else you need?


The other two are desktop machines with Core i7 desktop CPUs (the 2009 models), but Florian has more information about at least one of those computers. I am pretty sure that they had entirely different motherboards though (I forgot who the second i7 belonged to, hopefully Florian remembers).
Comment 9 Thomas Bächler 2010-04-12 21:20:52 UTC
Created attachment 25973 [details]
lspci on the Toshiba laptop
Comment 10 Thomas Bächler 2010-04-12 21:21:49 UTC
Created attachment 25974 [details]
dmidecode on the Toshiba laptop
Comment 11 Thomas Bächler 2010-04-12 21:22:45 UTC
Created attachment 25975 [details]
cpuinfo on the Toshiba laptop
Comment 12 Florian Pritz 2010-04-13 11:46:11 UTC
I'll also attach lspci, dmidecode and /proc/cpuinfo for the i7-920 machine.

> This smells like a BIOS problem to me.
>
I've just tried rebooting with kexec and it "fixes" the problem. I don't know if that excludes the BIOS though.
Comment 13 Florian Pritz 2010-04-13 11:47:18 UTC
Created attachment 25988 [details]
/proc/cpuinfo for i7-920
Comment 14 Florian Pritz 2010-04-13 11:47:59 UTC
Created attachment 25989 [details]
dmidecode for i7-920
Comment 15 Florian Pritz 2010-04-13 11:48:57 UTC
Created attachment 25990 [details]
lspci for i7-920
Comment 16 Andy B 2010-04-13 14:01:07 UTC
I have the same problem Florian described.

Here are my system specs:
CPU: Intel i7-860 (Core S1156 i7-860 BOX 4x2,80Ghz/256k 8MB-L3 QPI4.8)
Board: S1156 ASUS P7P55D LE
RAM: DDR3  8192MB 4x2 PC1333 G.Skill RipJaws CL9-9-9-24
Arch: x86_64
Distribution: Archlinux

I hope this helps.
Comment 17 Suresh B Siddha 2010-04-13 22:41:05 UTC
Can someone please provide dmesg after a resume and when this slowness problem is seen?

Florian, in the bug description you mentioned that you see one core at 100% and the rest betwen 0-10%. Can you please do an experiment of booting with maxcpus=1 and see if you encounter similar slowdown after suspend/resume? Thanks.
Comment 18 Florian Pritz 2010-04-14 15:42:19 UTC
With maxcpus=1 the system is even slower. 

Normally make -j12 uses around 700% cpu (maximum is 800%) after a reboot. After suspending it uses something between 200% and 500% and buildtime goes up from 10:30 to 12-25 minutes. 
I've built everything in a tmpfs so caches shouldn't matter.
Comment 19 Florian Pritz 2010-04-14 15:44:36 UTC
Created attachment 26000 [details]
dmesg after suspending 2 times

The low memory error in line 1033 doesn't happen on the other 2 systems so it can be ignored IMHO.
Comment 20 Suresh B Siddha 2010-04-15 00:30:37 UTC
Florian, Thanks for the log. During boot, kernel is already correcting e820 entries as the e820 entries given by the bios and MTRR entries are not in sync. And given that you are seeing issues with maxcpus=1 aswell, sounds like a bios issue to me. How much perf difference do you see with maxcpus=1 before and after suspend?

With maxcpus=1 and before/after suspend, can you please see if you see any difference in the output of "cat /proc/mtrr"

FWIW, I don't see similar slowdown on my Lenovo T410 core i5 laptop with latest Linus kernel.
Comment 21 Florian Pritz 2010-04-15 10:39:30 UTC
Oh sorry. I've been unclear there. maxcups=1 is slower than 8 cpus after suspend, but suspend with 1 doesn't show any difference. At first I thought you ment whether 1 core is as fast as 8 cores after suspend. (After suspend only 1 core is really used and the others are nearly unused).

Some statistics for readline (kernel needs too much time to test)
time are avg. over at least 3 runs; always built in a tmpfs with enough free RAM.

8 cores:
after reboot: 650-700% cpu usage; 2.1 sec time for make -j12
after suspend: 100-130% cpu usage; 8.5 sec time for make -j12

1 core:
after reboot: 91-95% cpu usage; 9.6 sec time for make -j1
after suspend: 91-95% cpu usage; 9.6 sec time for make -j1 (yes it's the same)

So 1 core doesn't show the bug, but with 8 cores it is there.

mtrr doesn't change and alwas looks like this:
reg00: base=0x000000000 (    0MB), size= 4096MB, count=1: write-back
reg01: base=0x0e0000000 ( 3584MB), size=  512MB, count=1: uncachable
reg02: base=0x0d0000000 ( 3328MB), size=  256MB, count=1: uncachable
reg03: base=0x100000000 ( 4096MB), size= 4096MB, count=1: write-back
reg04: base=0x200000000 ( 8192MB), size= 1024MB, count=1: write-back
reg05: base=0x230000000 ( 8960MB), size=  256MB, count=1: uncachable
Comment 22 Florian Pritz 2010-04-15 15:13:49 UTC
> FWIW, I don't see similar slowdown on my Lenovo T410 core i5 laptop with
> latest
Linus kernel.

Sill bugged here with v2.6.34-rc4-82-g250541f
Comment 23 Thomas Bächler 2010-04-15 19:11:10 UTC
Same test (readline build) with my laptop (see the specs above) with make -j5 (I didn't try the maxcpus=1 though). I have 2 Cores each with HT:

before suspend: 4.5s
after suspend:  9.2s

In the KDE systemloadviewer, you can clearly see how all 4 virtual cores are at 100% before the suspend, and only one is at 100% after suspend.

I still don't believe in a BIOS problem, as the same problem occurs here on an entirely different machine than Florian has. Kernel tested in my case is 2.6.33.2.
Comment 24 Thomas Bächler 2010-04-15 19:12:20 UTC
Created attachment 26022 [details]
dmesg after suspending once
Comment 25 Thomas Bächler 2010-04-18 13:36:10 UTC
The problem persists after upgrading to the latest ACPI/EC provided by Toshiba.
Comment 26 Suresh B Siddha 2010-04-20 00:38:14 UTC
Florian, Thomas: This now sounds more like a scheduler issue that is somehow triggered after suspend/resume. Can you please try few things like:

a) Try doing multiple compilations with j1, one on each core (by doing taskset) and compare performance before/after suspend. Atleast this should let us know if any particular core performance gets affected by suspend. If it is same, then most likely the problem will be in the scheduler.

b) Also, can you please try disabling different scheduler config options like SCHED_MC, SCHED_SMT, GROUP_SCHED options etc and see if it makes any difference in the behavior.

I still don't have any luck in reproducing the issue.
Comment 27 Florian Pritz 2010-04-20 15:59:30 UTC
(In reply to comment #26)
> a) Try doing multiple compilations with j1, one on each core (by doing
> taskset)
> and compare performance before/after suspend. Atleast this should let us know
> if any particular core performance gets affected by suspend. If it is same,
> then most likely the problem will be in the scheduler.

All have the same speed before/after.

> b) Also, can you please try disabling different scheduler config options like
> SCHED_MC, SCHED_SMT, GROUP_SCHED options etc and see if it makes any
> difference
> in the behavior.

Doesn't help.

Though while testing I noticed that the bug sometimes (like 50:50 I think) vanishes after a few make runs or after a couple of minutes after the first suspend. When I suspend again this doesn't happen anymore (maybe bad luck). Sadly I can't find a way to reproduce it.

Can I somehow get verbose CPU scheduler logging in dmesg?
Comment 28 Stephan Eicher 2010-04-20 16:15:33 UTC
Same problem here. My maschine is a HP DV8-1050eg (notebook)

if you need anymore beside lspci and dmidecode just tell me.



output of lspci:

00:00.0 Host bridge: Intel Corporation Core Processor DMI (rev 11)
00:03.0 PCI bridge: Intel Corporation Core Processor PCI Express Root Port 1 (rev 11)
00:08.0 System peripheral: Intel Corporation Core Processor System Management Registers (rev 11)
00:08.1 System peripheral: Intel Corporation Core Processor Semaphore and Scratchpad Registers (rev 11)
00:08.2 System peripheral: Intel Corporation Core Processor System Control and Status Registers (rev 11)
00:08.3 System peripheral: Intel Corporation Core Processor Miscellaneous Registers (rev 11)
00:10.0 System peripheral: Intel Corporation Core Processor QPI Link (rev 11)
00:10.1 System peripheral: Intel Corporation Core Processor QPI Routing and Protocol Registers (rev 11)
00:1a.0 USB Controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 05)
00:1b.0 Audio device: Intel Corporation 5 Series/3400 Series Chipset High Definition Audio (rev 05)
00:1c.0 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 1 (rev 05)
00:1c.1 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 2 (rev 05)
00:1c.4 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 5 (rev 05)
00:1c.7 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 8 (rev 05)
00:1d.0 USB Controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation Mobile 5 Series Chipset LPC Interface Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 5 Series/3400 Series Chipset SMBus Controller (rev 05)
01:00.0 VGA compatible controller: nVidia Corporation GeForce GT 230M (rev a2)
01:00.1 Audio device: nVidia Corporation High Definition Audio Controller (rev a1)
02:00.0 Network controller: Intel Corporation PRO/Wireless 5100 AGN [Shiloh] Network Connection
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
04:00.0 FireWire (IEEE 1394): JMicron Technology Corp. IEEE 1394 Host Controller
04:00.1 System peripheral: JMicron Technology Corp. SD/MMC Host Controller
04:00.2 SD Host controller: JMicron Technology Corp. Standard SD Host Controller
04:00.3 System peripheral: JMicron Technology Corp. MS Host Controller
04:00.4 System peripheral: JMicron Technology Corp. xD Host Controller
ff:00.0 Host bridge: Intel Corporation Core Processor QuickPath Architecture Generic Non-Core Registers (rev 04)
ff:00.1 Host bridge: Intel Corporation Core Processor QuickPath Architecture System Address Decoder (rev 04)
ff:02.0 Host bridge: Intel Corporation Core Processor QPI Link 0 (rev 04)
ff:02.1 Host bridge: Intel Corporation Core Processor QPI Physical 0 (rev 04)
ff:03.0 Host bridge: Intel Corporation Core Processor Integrated Memory Controller (rev 04)
ff:03.1 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Target Address Decoder (rev 04)
ff:03.4 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Test Registers (rev 04)
ff:04.0 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 0 Control Registers (rev 04)
ff:04.1 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 0 Address Registers (rev 04)
ff:04.2 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 0 Rank Registers (rev 04)
ff:04.3 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 0 Thermal Control Registers (rev 04)
ff:05.0 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 1 Control Registers (rev 04)
ff:05.1 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 1 Address Registers (rev 04)
ff:05.2 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 1 Rank Registers (rev 04)
ff:05.3 Host bridge: Intel Corporation Core Processor Integrated Memory Controller Channel 1 Thermal Control Registers (rev 04)


output of dmidecode:

# dmidecode 2.10
SMBIOS 2.6 present.
37 structures occupying 1648 bytes.
Table at 0x000EA900.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Hewlett-Packard
        Version: F.15
        Release Date: 12/28/2009
        ROM Size: 1536 kB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Japanese floppy for NEC 9800 1.2 MB is supported (int 13h)
                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                8042 keyboard services are supported (int 9h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
        BIOS Revision: 0.27
        Firmware Revision: 53.36

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: Hewlett-Packard
        Product Name: HP Pavilion dv8 Notebook PC
        Version: 039D200000241220001020000
        Serial Number: CNF941F1HS
        UUID: 39464E43-3134-3146-4853-00269E5BC4E6
        Wake-up Type: Power Switch
        SKU Number: VJ747EA#ABD
        Family: 103C_5335KV

Handle 0x0002, DMI type 2, 16 bytes
Base Board Information
        Manufacturer: Hewlett-Packard
        Product Name: 7001
        Version: 35.24
        Serial Number: CNF941F1HS
        Asset Tag: Base Board Asset Tag
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: Base Board Chassis Location
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
        Manufacturer: Hewlett-Packard
        Type: Notebook
        Lock: Not Present
        Version: N/A
        Serial Number: None
        Asset Tag:                     
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: None
        OEM Information: 0x00000212
        Height: Unspecified
        Number Of Power Cords: 1
        Contained Elements: 0

Handle 0x0004, DMI type 9, 17 bytes
System Slot Information
        Designation: J5C1
        Type: x16 PCI Express x16
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x0005, DMI type 9, 17 bytes
System Slot Information
        Designation: J6C1
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x0006, DMI type 9, 17 bytes
System Slot Information
        Designation: J6C2
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x0007, DMI type 9, 17 bytes
System Slot Information
        Designation: J6D2
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x0008, DMI type 9, 17 bytes
System Slot Information
        Designation: J7C1
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x0009, DMI type 9, 17 bytes
System Slot Information
        Designation: J7D2
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x000A, DMI type 9, 17 bytes
System Slot Information
        Designation: J8C2
        Type: x16 PCI Express x16
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x000B, DMI type 9, 17 bytes
System Slot Information
        Designation: J8C1
        Type: x1 PCI Express x1
        Current Usage: Available
        Length: Other
        Characteristics:
                PME signal is supported
                Hot-plug devices are supported
        Bus Address: 0000:00:1f.7

Handle 0x000C, DMI type 10, 6 bytes
On Board Device Information
        Type: Video
        Status: Enabled
        Description: Video Graphics Controller

Handle 0x000D, DMI type 10, 6 bytes
On Board Device Information
        Type: Ethernet
        Status: Enabled
        Description: Realtek Lan Controller

Handle 0x000E, DMI type 11, 5 bytes
OEM Strings
        String 1: $HP$
        String 2: LOC#ABD
        String 3: ABS 70/71 79 7A 7B 7C
        String 4: CNB1 039D200000241220001020000

Handle 0x000F, DMI type 12, 5 bytes
System Configuration Options
        Option 1: String1 for Type12 Equipment Manufacturer
        Option 2: String2 for Type12 Equipment Manufacturer
        Option 3: String3 for Type12 Equipment Manufacturer
        Option 4: String4 for Type12 Equipment Manufacturer

Handle 0x0010, DMI type 15, 29 bytes
System Event Log
        Area Length: 0 bytes
        Header Start Offset: 0x0000
        Data Start Offset: 0x0000
        Access Method: General-purpose non-volatile data functions
        Access Address: 0x0000
        Status: Valid, Not Full
        Change Token: 0x12345678
        Header Format: OEM-specific
        Supported Log Type Descriptors: 3
        Descriptor 1: POST memory resize
        Data Format 1: None
        Descriptor 2: POST error
        Data Format 2: POST results bitmap
        Descriptor 3: Log area reset/cleared
        Data Format 3: None

Handle 0x0011, DMI type 21, 7 bytes
Built-in Pointing Device
        Type: Touch Pad
        Interface: PS/2
        Buttons: 4

Handle 0x0012, DMI type 22, 26 bytes
Portable Battery
        Location: Primary
        Manufacturer: BP44-SMP
        Name: GA04063
        Chemistry: Lithium Ion
        Design Capacity: 4400 mWh
        Design Voltage: 14400 mV
        SBDS Version: 3.1
        Maximum Error: 1%
        SBDS Serial Number: 0088
        SBDS Manufacture Date: 2009-09-22
        OEM-specific Information: 0xFFFFFFFF

Handle 0x0013, DMI type 32, 20 bytes
System Boot Information
        Status: No errors detected

Handle 0x0014, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: NVIDIA Video Graphics Controller
        Type: Video
        Status: Enabled
        Type Instance: 1
        Bus Address: 0000:01:00.0

Handle 0x0015, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: Intel(R) WiFi Link 5100 AGN
        Type: Ethernet
        Status: Enabled
        Type Instance: 1
        Bus Address: 0000:02:00.0

Handle 0x0016, DMI type 16, 15 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: None
        Maximum Capacity: 8 GB
        Error Information Handle: No Error
        Number Of Devices: 2

Handle 0x0017, DMI type 17, 28 bytes
Memory Device
        Array Handle: 0x0016
        Error Information Handle: 0x0018
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 2048 MB
        Form Factor: SODIMM
        Set: None
        Locator: Bottom - Slot 1
        Bank Locator: BANK 0
        Type: <OUT OF SPEC>
        Type Detail: Synchronous
        Speed: 1067 MHz
        Manufacturer: Micron
        Serial Number: FD129AED
        Asset Tag: Unknown
        Part Number: 16JSF25664HZ-1G1F1
        Rank: Unknown

Handle 0x0018, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x0019, DMI type 20, 19 bytes
Memory Device Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x0007FFFFFFF
        Range Size: 2 GB
        Physical Device Handle: 0x0017
        Memory Array Mapped Address Handle: 0x001E
        Partition Row Position: 2
        Interleave Position: 1
        Interleaved Data Depth: 1

Handle 0x001A, DMI type 17, 28 bytes
Memory Device
        Array Handle: 0x0016
        Error Information Handle: 0x001B
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 2048 MB
        Form Factor: SODIMM
        Set: None
        Locator: Bottom - Slot 2
        Bank Locator: BANK 1
        Type: <OUT OF SPEC>
        Type Detail: Synchronous
        Speed: 1067 MHz
        Manufacturer: Micron
        Serial Number: FD129AEE
        Asset Tag: Unknown
        Part Number: 16JSF25664HZ-1G1F1
        Rank: Unknown

Handle 0x001B, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x001C, DMI type 20, 19 bytes
Memory Device Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x0007FFFFFFF
        Range Size: 2 GB
        Physical Device Handle: 0x001A
        Memory Array Mapped Address Handle: 0x001E
        Partition Row Position: 2
        Interleave Position: 2
        Interleaved Data Depth: 1

Handle 0x001D, DMI type 18, 23 bytes
32-bit Memory Error Information
        Type: OK
        Granularity: Unknown
        Operation: Unknown
        Vendor Syndrome: Unknown
        Memory Array Address: Unknown
        Device Address: Unknown
        Resolution: Unknown

Handle 0x001E, DMI type 19, 15 bytes
Memory Array Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x000FFFFFFFF
        Range Size: 4 GB
        Physical Array Handle: 0x0016
        Partition Width: 0

Handle 0x001F, DMI type 4, 42 bytes
Processor Information
        Socket Designation: CPU
        Type: Central Processor
        Family: Core 2 Duo
        Manufacturer: Intel(R) Corporation
        ID: E5 06 01 00 FF FB EB BF
        Signature: Type 0, Family 6, Model 30, Stepping 5
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                DS (Debug store)
                ACPI (ACPI supported)
                MMX (MMX technology supported)
                FXSR (Fast floating-point save and restore)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                SS (Self-snoop)
                HTT (Hyper-threading technology)
                TM (Thermal monitor supported)
                PBE (Pending break enabled)
        Version: Intel(R) Core(TM) i7 CPU       Q 720  @ 1.60GHz
        Voltage: 1.2 V
        External Clock: 1066 MHz
        Max Speed: 1600 MHz
        Current Speed: 1600 MHz
        Status: Populated, Enabled
        Upgrade: ZIF Socket
        L1 Cache Handle: 0x0023
        L2 Cache Handle: 0x0022
        L3 Cache Handle: 0x0020
        Serial Number: Not Specified
        Asset Tag: FFFF
        Part Number: Not Specified
        Core Count: 4
        Core Enabled: 4
        Thread Count: 8
        Characteristics:
                64-bit capable

Handle 0x0020, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L3 Cache
        Configuration: Enabled, Not Socketed, Level 3
        Operational Mode: Write Through
        Location: Internal
        Installed Size: 6144 kB
        Maximum Size: 6144 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: Other

Handle 0x0021, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L1 Cache
        Configuration: Enabled, Not Socketed, Level 1
        Operational Mode: Write Through
        Location: Internal
        Installed Size: 32 kB
        Maximum Size: 32 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Data
        Associativity: 8-way Set-associative

Handle 0x0022, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L2 Cache
        Configuration: Enabled, Not Socketed, Level 2
        Operational Mode: Write Through
        Location: Internal
        Installed Size: 256 kB
        Maximum Size: 256 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: 8-way Set-associative

Handle 0x0023, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L1 Cache
        Configuration: Enabled, Not Socketed, Level 1
        Operational Mode: Write Through
        Location: Internal
        Installed Size: 32 kB
        Maximum Size: 32 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Instruction
        Associativity: 4-way Set-associative

Handle 0x0024, DMI type 127, 4 bytes
End Of Table
Comment 29 Stephan Eicher 2010-04-20 16:18:41 UTC
Created attachment 26063 [details]
lspci laptop HP DV8-1050eg
Comment 30 Stephan Eicher 2010-04-20 16:19:23 UTC
Created attachment 26064 [details]
dmidecode of HP DV8-1050eg
Comment 31 Thomas Bächler 2010-04-20 16:43:52 UTC
Stephan, which distribution and kernel?
Comment 32 Stephan Eicher 2010-04-20 16:48:39 UTC
Distribution:
Arch Linux stable

Kernel:
Linux Linux 2.6.33-ARCH #1 SMP PREEMPT Sun Apr 4 10:27:30 CEST 2010 x86_64 Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz GenuineIntel GNU/Linux
Comment 33 Thomas Bächler 2010-04-20 16:51:38 UTC
Okay, all affected people are German and use Arch Linux - that may be because Florian is urging everyone on freenode.net/#archlinux.de to confirm his bug. Or it may be because we have something weird in our kernel configuration that causes the bug that nobody else has.
Comment 34 Rafael J. Wysocki 2010-04-20 18:45:10 UTC
Well, can any of you try 2.6.34-rc5 from kernel.org, pretty please?
Comment 35 Suresh B Siddha 2010-04-20 19:12:49 UTC
Florian, Thomas etc, can you please collect the scheduler trace output when you see this problem.

# mount -t debugfs debugfs /sys/kernel/debug

# echo sched_switch > /sys/kernel/debug/tracing/current_tracer

# <run your workload>

# echo  0 > /sys/kernel/debug/tracing/trace; sleep 5; cat /sys/kernel/debug/tracing/trace > /tmp/sched_switch_tracing_data.txt

It looks to be some scheduling anomaly. Also adding PeterZ to see if he has seen any similar issues.
Comment 36 Florian Pritz 2010-04-20 19:57:43 UTC
Created attachment 26069 [details]
1st Scheduler trace (long)
Comment 37 Florian Pritz 2010-04-20 19:59:11 UTC
Created attachment 26070 [details]
2nd Scheduler trace (short)

Both traces were made when the bug occurred. (After suspending once)
Comment 38 Florian Pritz 2010-04-20 20:09:36 UTC
(In reply to comment #34)
> Well, can any of you try 2.6.34-rc5 from kernel.org, pretty please?

2.6.34-rc5-marin-00023-g05ce7bf still the same problem.
Comment 39 Suresh B Siddha 2010-04-20 20:13:40 UTC
Florian, I meant to say "run your workload in the background". Please ensure that the workload runs for the complete duration of trace collection.
Comment 40 Florian Pritz 2010-04-20 20:19:58 UTC
Created attachment 26071 [details]
Scheduler trace while building kernel

Hope this one's better. I ran make -j12 on a kernel tree and cpu usage was somewhere like 300-400%.
Comment 41 Suresh B Siddha 2010-04-21 23:38:02 UTC
Florian, I see some disk IO happening in your trace.

# cat sched_switch_tracing_data.txt | grep :D | wc -l
469

When you said cpu usage is 300-400%, how much of the rest is reported as iowait time vs idle time?

In comment #21, you mentioned the numbers were average of 3 runs. Are those 3 runs done after a resume or you did 3 resumes and one run after each resume?

Also, do you see a similar problem by offlining all the non boot cpus (by doing echo 0 > /sys/devices/system/cpu/cpuX/online) and onlining them (instead of suspend/resume)?
Comment 42 Florian Pritz 2010-04-22 12:02:33 UTC
(In reply to comment #41)
> Florian, I see some disk IO happening in your trace.
> 
> # cat sched_switch_tracing_data.txt | grep :D | wc -l
> 469
> 
> When you said cpu usage is 300-400%, how much of the rest is reported as
> iowait
> time vs idle time?

That's what top shows when compiling:
Cpu(s): 46.4%us,  4.9%sy,  0.0%ni, 48.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

vmstat shows 0 most of the time (sometimes up to 50) in the io section.

> In comment #21, you mentioned the numbers were average of 3 runs. Are those 3
> runs done after a resume or you did 3 resumes and one run after each resume?

3 runs of make; suspend; 3 runs of make again (without suspending between them)

> Also, do you see a similar problem by offlining all the non boot cpus (by
> doing
> echo 0 > /sys/devices/system/cpu/cpuX/online) and onlining them (instead of
> suspend/resume)?

No, that doesn't affect performance.
Comment 43 how 2010-06-30 12:15:38 UTC
Same problem here with kernel 2.6.34. 
Only one of the core is used after wake up from hibernate or suspend.

May be is hyperthreading problem. I cannot disable ht with kernel boot options to test.

Have any of you tried disabling ht in kernel to see if that works?

Somehow it randomly went back to normal performance long after wakeup from suspend. But thats no good.
Comment 44 Florian Pritz 2010-06-30 12:47:48 UTC
(In reply to comment #43)
> Same problem here with kernel 2.6.34. 

Which distribution do you use and how do you suspend?
Comment 45 how 2010-07-01 08:46:51 UTC
Linux Mint 9. I know is not distribution dependent because I read elsewhere on the web also reported with Debian.

Acer Aspire laptop. I suspend by using gnome-power-manager. I test it with stress -c 4. Only one of the core, but can be any one is used.
Comment 46 Florian Pritz 2010-07-01 22:28:54 UTC
(In reply to comment #43)
> May be is hyperthreading problem. I cannot disable ht with kernel boot
> options
> to test.
> Have any of you tried disabling ht in kernel to see if that works?
> Somehow it randomly went back to normal performance long after wakeup from
> suspend. But thats no good.

Okay then that's not only me. I still believe though that even if stress uses all 8 cores, it's slower than it should be. Think I once did the readline test when that happened and it still took ~9secs (maybe 6-8, but definitely not 2).

Also HT is no kernel settings, but rather a BIOS option.
Comment 47 how 2010-07-02 04:03:54 UTC
Well, unfortunately there is no HT option in my bios.

What I found at the moment might give some lead to the bug.
stress -c 4 -> only 1 core, randomly any core.
stress -c 100 -> 2 cores are running, that is in HT 1 core but 2 logical cores are running.
stress -c 400 -> 4 cores are running.

What is should have been is spawn 4 workers, 4 cores are used. But instead needs huge amount of workers needed to trigger all 4 cores.
Comment 48 how 2010-07-03 10:03:09 UTC
Now, I just tried 2.6.31 from ubuntu kernel ppa
It seemed this one works fine. I suspended 3 times all worked fine. I tested using python multiprocessing. Florian you are right, even Stress uses all cores it still runs slow with other kernels.
Comment 49 how 2010-07-08 11:16:51 UTC
The kernel 2.6.31 from karmic does not work.

Please try the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.31.13-karmic/ This one works with suspend. I observed at times still there is a slight delay for all the core to kick in, but much better than 2.6.32, 2.6.33, 2.6.34 and 2.6.35
Comment 50 Chris Frost 2010-07-16 06:34:38 UTC
I see similar (but not identical, see below) behavior with a Core i7-930
CPU and EVGA 141-BL-E757-TR motherboard, running either of the 2.6.34.1
from kernel.org and the 2.6.32 in x86-64 Ubuntu 10.04. Both are x86-64.
The kernel only automatically schedules processes on the first one or two
cores after resuming from a suspend to RAM or disk. The machine does not
exhibit this undesired behavior when running the 2.6.31 kernel cited above
in comment #49.

I see two differences in behavior from earlier comments:
* After resuming, setting the appropriate CPU affinities allows processes
  to run on all cores and HT units. (Florian saw no change with taskset.)
* I have seen the kernel automatically use both only one or two cores.
  It seems to vary. (Earlier comments only mention one core.)

I'll also attach a simplified workload that experiences this poor behavior:
the program spins in an infinite loop and prints the elapsed time every
N iterations. The expected behavior is that, when running only one process,
that process will see the same performance as each process does when four
run concurrently. This is true for me before suspending and when manually
setting CPU affinity after resuming, but not after resuming when the kernel
automatically schedules processes.  These processes touch a smaller cross
section of the kernel than a kernel build does; in particular, they cause
no disk I/O.

In addition to seeing the iterations per second drop when adding the second
or third spin process after resuming, top shows a drop in percent CPU usage
for each process (a drop from 100% to <=50%). /proc/cpuinfo shows
corresponding data for core frequencies: before suspending all four cores
are at 2.8GHz (the maximum reported available) and after only one or two
cores are at 2.8GHz (the others are at their lowest frequency, 1.6GHz).

FWIW, I also see unexpected results when using more than four cores, using
HT. I've not looked into these enough to describe them well, but I'll do
this if it might be helpful.
Comment 51 Chris Frost 2010-07-16 06:37:22 UTC
Created attachment 27128 [details]
Simplified workload: a process that spins and reports its speed
Comment 52 Mans Rullgard 2010-07-17 11:04:26 UTC
I'm getting this on a Sony VAIO Z1 laptop with i5-M540 CPU, BIOS version R2074C3, vanilla 2.6.34.1 kernel.  Automatic scheduling places everything on the same CPU (which one varies), although I can force them elsewhere with taskset.
Comment 53 how 2010-07-17 11:58:36 UTC
schedule multi-threaded task to run on all CPU with taskset still runs on one CPU. CPU affinity does not work with these kernels other than the version in comment #49.
Comment 54 Mans Rullgard 2010-07-19 17:53:19 UTC
Just be clear, the processes move around between cores, but only one at a time has significant load.
Comment 55 Mans Rullgard 2010-07-19 17:59:42 UTC
One more observation: an explicit "taskset 3 make -j4" loads cpu0 and cpu1 fully, while "taskset 15 ..." (4 virtual cores) loads only cpu0 and cpu2, and "taskset 7" results in load moving erratically between the first 3 cores, never fully loading all at the same time.
Comment 56 ChriS 2010-07-23 18:13:42 UTC
Created attachment 27226 [details]
add.c (test program)

I have the same problem on a

  Thinkpad W510, Core i7 Q 820  @ 1.73GHz
  Distribution: Debian (testing)
  Linux poincare 2.6.34.1 #1 SMP Sun Jul 18 00:24:31 CEST 2010 x86_64 GNU/Linux

but the funny think is that suspending results in faster execution after!  It would be nice to have the increased speed before suspending as well!

Here are times for a compilation of the kernel and a run of the
attached program.

make -j12 kernel 2.6.34.1
before suspend:
  1427.02user 99.36system 3:30.68elapsed 724%CPU (0avgtext+0avgdata 351520maxresident)k
  0inputs+0outputs (90major+29658591minor)pagefaults 0swaps
after suspend:
  1164.72user 83.47system 4:00.93elapsed 518%CPU (0avgtext+0avgdata 350176maxresident)k
  0inputs+0outputs (15major+29658285minor)pagefaults 0swaps

/usr/bin/time ./add
before suspend:
  3.62user 0.00system 0:03.63elapsed 99%CPU (0avgtext+0avgdata 2032maxresident)k
  0inputs+0outputs (0major+171minor)pagefaults 0swaps
after suspend:
  1.47user 0.00system 0:01.47elapsed 99%CPU (0avgtext+0avgdata 2032maxresident)k
  0inputs+0outputs (0major+170minor)pagefaults 0swaps
Comment 57 ChriS 2010-08-05 12:24:19 UTC
The bug is still present in kernel 2.6.35.
Comment 58 how 2010-08-05 12:36:00 UTC
of course is present in 2.6.35, because they dont know what cause that problem and there is no fix for now.
Comment 59 Suresh B Siddha 2010-08-07 01:27:47 UTC
Created attachment 27368 [details]
Patch to root cause the issue

All those who had encountered this issue, please try the attached patch (for 2.6.35). Post 2.6.35 has some changes in this area and as such doesn't apply as it is to the current mainline.

It looks like update_group_power() in kernel/sched_fair.c is broken and thus causing the issue. If you folks can confirm that the attached patch indeed solves the problem, then I can work with Peter Zijlstra to come up with the appropriate fix for the broken update_group_power() code.
Comment 60 how 2010-08-07 03:00:30 UTC
Great ! lets try out the patch.

Does anyone know how to compile in deb pkg that comes with the Vanilla Kernel to only support subset of modules like the one comes with Ubuntu distro. If I do make deb-pkg it compiles all modules.
Comment 61 how 2010-08-07 08:02:35 UTC
Awesome, the patch works !! only one line of code.
Comment 62 how 2010-08-07 09:37:49 UTC
After the patch, now activate the suspend takes forever to sleep. and after wakeup the desktop graphic is really slow
Comment 63 ChriS 2010-08-07 10:50:56 UTC
Your patch works in the sense that all CPUs are equally used before and after suspend:

$ make -j12 # kernel 2.6.35
* before suspend:
1476.15user 104.60system 3:32.17elapsed 745%CPU (0avgtext+0avgdata 351760maxresident)k
0inputs+0outputs (9major+31032049minor)pagefaults 0swaps
* after suspend:
1477.85user 102.40system 3:28.08elapsed 759%CPU (0avgtext+0avgdata 355168maxresident)k
0inputs+0outputs (0major+31031949minor)pagefaults 0swaps

However, for the "add" program attached above, I still see a significant speed difference:

$ ./add
* before suspend:
3.59user 0.00system 0:03.60elapsed 99%CPU (0avgtext+0avgdata 2032maxresident)k
0inputs+0outputs (0major+169minor)pagefaults 0swaps
* after suspend:
1.44user 0.12system 0:01.57elapsed 99%CPU (0avgtext+0avgdata 2032maxresident)k
0inputs+0outputs (0major+169minor)pagefaults 0swaps

As a related matter, it would be nice if the scheduler was aware of hyperhreading/turboboost more than it is now as selecting only half the CPUs make the compilation way faster:

$ taskset -c 0,2,4,6 make -j12
* before suspend:
870.01user 73.80system 4:10.64elapsed 376%CPU (0avgtext+0avgdata 354960maxresident)k
0inputs+0outputs (184major+31031346minor)pagefaults 0swaps
* after suspend:
881.20user 64.93system 4:06.67elapsed 383%CPU (0avgtext+0avgdata 354960maxresident)k
0inputs+0outputs (0major+31031107minor)pagefaults 0swaps
Comment 64 Florian Pritz 2010-08-07 11:24:28 UTC
Building readline takes the same time before and after suspend, stress -c 8 uses all 8 virtual CPUs. 

Tested with 2.6.35(In reply to comment #62)
> After the patch, now activate the suspend takes forever to sleep. and after
> wakeup the desktop graphic is really slow
Can't reproduce that here (nvidia graphics with the prop. driver), but I believe that will be gone when the real fix is done.

Tested with 2.6.35, suspended twice
Comment 65 how 2010-08-07 22:33:56 UTC
(In reply to comment #64)
> Building readline takes the same time before and after suspend, stress -c 8
> uses all 8 virtual CPUs. 
> 
> Tested with 2.6.35(In reply to comment #62)
> > After the patch, now activate the suspend takes forever to sleep. and after
> > wakeup the desktop graphic is really slow
> Can't reproduce that here (nvidia graphics with the prop. driver), but I
> believe that will be gone when the real fix is done.
> 
> Tested with 2.6.35, suspended twice

Works fine only for the first 2 or 3 suspends but after that takes longer for screen to dim before actually sleeps. The graphic (nvidia) dont really work, relogin fixes it. I guess this is what could happen if update_group_power is disabled,thats what the patch does.
Comment 66 Suresh B Siddha 2010-08-10 07:56:07 UTC
Thanks all for testing.

reg comment #63

Chris, You need to look at the elapsed time for kernel compilation and you can see using HT improves the performance. And related to "add" program, I suspect, P-states or something else is playing a role and is a different issue altogether.

reg comment #65

wong, I am not sure what you are seeing after second or third suspend is because of the test patch. I will post another patch that will be a more acceptable solution. It will be great if you can test it again. If you still see the issue with the patch, try doing the same test with "maxcpus=1". if you still see the issue, then it is a different issue.
Comment 67 Suresh B Siddha 2010-08-10 08:01:00 UTC
Created attachment 27396 [details]
Proposed patch to fix the issue

All, the changelog of the attached patch explains what the issue is and how it is addressed. This patch is on top of latest linus git (post 2.6.35). Please give it a try and let me know if I have your Ack. I will post it to lkml in the morning. Thanks.
Comment 68 how 2010-08-10 10:07:59 UTC
I got 2 different weird results. one with all CPUS and the other is maxcpus=1 setting.

with all cpus:
sleep is ok. wakeup stuck with cpu up and no wireless light and screen still off for about 1min may be 30s, then everything comes on as normal. The multithread seem to work with all cpu used but somehow decoding H264 is very choppy. CPUs still doesnt function as normal.

with maxcpus=1:
sleep is ok. wakeup stuck as above, but with wireless light on before wakes up completely. Decoding H264 is normal. Lightsmark2008 is normal. Seems cpu function is normal.

I am totally confused at this stage. Multithread is working doesnt mean cpu is working to full potential.
Comment 69 ChriS 2010-08-10 11:25:24 UTC
Suresh B Siddha: of course you are right about HT, I was struck by the difference in user time and posted too fast.

About you patch: the machine freezes during wakeup, it never comes out of sleep.

For the "add" program: I will open a new bug unless you tell me there is a better course of action.
Comment 70 Suresh B Siddha 2010-08-11 05:45:46 UTC
Ok. I am able to see the hang with my patch in comment #67. For now please ignore it. It seems like the after resume TSC is set to a large value on the thinkpad 510 and perhaps this is the fundamental reason why we are seeing this only on specific laptops. Let me getback with an appropriate fix for this in a day. Thanks.
Comment 71 Suresh B Siddha 2010-08-11 19:24:17 UTC
Created attachment 27410 [details]
Second attempt at fixing the issue

A simple patch to address this issue. Please retest again and let me know your Acks. Thanks.
Comment 72 how 2010-08-11 23:50:57 UTC
second patch tested.

wakeup back to normal. Multithread works. H264 cpu decoding dont work ( many pauses). Lightsmark2008 with nvidia dont work(runs slow).
Comment 73 Suresh B Siddha 2010-08-12 00:17:00 UTC
wong, thanks. when you say multithread works, which workload are you using?

And H264 decoding and lightsmark2008 are those single threaded workloads that you are running? And if so, after resume can you invoke them on different cpu's (using taskset) and see if you see different performance. It sounds like this is a different issue and not related to the scheduler issue that this bug originally started with. And also what is your system configuration?
Comment 74 how 2010-08-12 03:36:52 UTC
With multithread test I used python multithread module to compute sum of prime numbers. In normal condition the program finishes in 5s. If it is run on single cpu/thread or the suspend issue it takes 9s to finish. With your new patch it also finishes in 5s

H264 and Lightsmark are single threads because they perform the same when maxcpus=1 as in multicore

Before your patches H264 decoding and Lightsmark never worked after suspend. I still believe is related to the scheduler. With maxcpus=1 it all works as mentioned in #68

taskset does not improve the situation. Although H264 decoding less pauses after the patch. Lightsmark works randomly.

System Configuration is Aspire notebook i5 430m
Comment 75 how 2010-08-12 03:40:38 UTC
Overall I think the second patch improved system somewhat.
Comment 76 psa 2010-08-12 07:35:56 UTC
Hello everyone. I'm using Vaio vpcs11x9r with core i3 and nvidia 310m on board. I have the same problems on the architecture of x86. I applied the patch to the kernel of 2.6.35, but the changes after pm-suspend does not watch.
Or, maybe this thread only for i5-i7 processors?

How can I help in solving the problem?

Problems with graphics (and video) appear after the actions described here: http://www.nvnews.net/vbulletin/showpost.php?p=2300624&postcount=2 - perhaps this is due to the drivers for nvidia video cards?
Comment 77 ChriS 2010-08-12 07:40:51 UTC
I am happy to report that the patch posted in comment #71 works for me (Thinkpad W510).  Here are the results for the compilation of kernel 2.6.35:

$ make -j 12
* before suspend:
1477.57user 102.39system 3:33.59elapsed 739%CPU (0avgtext+0avgdata 355184maxresident)k
0inputs+0outputs (53major+31049166minor)pagefaults 0swaps
* after suspend:
1483.64user 102.20system 3:28.23elapsed 761%CPU (0avgtext+0avgdata 354672maxresident)k
0inputs+0outputs (0major+31049312minor)pagefaults 0swaps

Thanks for your work!
Comment 78 ChriS 2010-08-12 07:53:52 UTC
BTW, I have a
01:00.0 VGA compatible controller: nVidia Corporation GT216 [Quadro FX 880M] (rev a2)
with, unfortunately, the proprietary driver (needed for using a data-projector) and it shows none of the problems of comment #76.
Comment 79 psa 2010-08-12 08:06:10 UTC
Maybe I'm wrong, but if I quickly recover from suspend - then no problem. If I leave laptop in suspend mode for 30-40 minutes - then the performance problems occur.
Can affect suspend time for anything?
Comment 80 how 2010-08-12 08:43:49 UTC
ChriS: I believe the make -j works just the same as what I tested with my python program in #72. But I dont think this is the same as decoding H264 cpu calculations. Can you please confirm decoding H264 indeed works on your notebook also? I used 1080p resolution mplayer.
Comment 81 how 2010-08-12 08:46:18 UTC
psa: what do you use to test the patch?
Comment 82 psa 2010-08-12 14:28:56 UTC
For testing I use debian sid:
1. `make clean && time make-j4`, kernel 2.6.35
2. for visual assessment - guitar pro 6 with wine 1.3
3. 1280x720 H264 video with mplayer -vo xv

Again, carefully testing the kernel, I found that:

before suspend (both cores showed similar results):
1. make -j4  3739,31s user 256,01s system 345% cpu 19:16,49 total
2. guitar pro 6 opening "4 seasons Vivaldi" for 5 seconds
3. CPU usage even at very complex scenes is less than 60%.

after suspend:

Vanilla kernel:
1. make -j4  3189,52s user 223,88s system 253% cpu 22:25,21 total - In this case the maximum load reached only 2 processors and a feeling that the real work only physical processors without HT mode.
2. guitar pro 6 opening "4 seasons" for 35 seconds, while running a processor from four and 90% of his busy process "/usr/bin/X :0 vt7 -br -nolisten tcp -auth /var/run/xauth/A:0-VvOiXb" overall responsiveness of the shell drops sharply.
3. a very big problem with playback. Responsiveness to rewind - delayed by 2 seconds. When you leave a little hanging c twitching pictures, but eventually mplayer ends.
After 5 minutes of the first compilation I tried the second time:
1b. make -j4  3800,78s user 265,19s system 357% cpu 18:56,15 total - for the second time all processors are involved - the system as if finally woke up ...
2b. the same delay in wine
3b. same problem with the video.

Patched kernel:
1. make -j4  3778,02s user 254,40s system 362% cpu 18:32,77 total - normal
2. guitar pro 6 opening "4 seasons" for 25 seconds, while the problems are the same as in the vanilla kernel.
3. OK, but the load on the system much more a single processor reaches 90-100% of the active change of frame. Responsiveness of the shell while playing a few below.
Comment 83 psa 2010-08-12 14:31:50 UTC
The patch really fix the problem in the kernel - other errors seem to occur in the environment.
Comment 84 ChriS 2010-08-12 15:09:02 UTC
How Wong: I have no problem in viewing FHD H.264 (with mplayer and vlc).
Comment 85 how 2010-08-13 09:14:04 UTC
Chris: thanks for testing. 

I am not sure whether is i7 and i5 difference. I still have problem with the patch with decoding H264.

psa: Did the patch fix your decoding H264 responsiveness? I still have have the responsiveness issue just like yours before the patch. After the patch I still have it.
Comment 86 psa 2010-08-13 10:10:32 UTC
The patch is partly solved the problem with the video, but not until the end. I think that is also a problem in Xorg or videocard driver.
Comment 87 how 2010-08-13 10:52:47 UTC
The deooding problem only happens after suspend. If it is not related to scheduler than why does maxcpus=1 works with or without the patch? 

I am not sure if it has any thing to do with this? although this one is to do with x264 encoding
http://www.stationstops.com/2010/01/25/linux-kernel-issues-with-cpu-scheduling-and-x264/
Comment 88 Suresh B Siddha 2010-08-20 00:30:48 UTC
Based on PeterZ's feedback I posted another version that fixes the scheduler issue triggered by suspend/resume.

http://marc.info/?l=linux-kernel&m=128226270213414&w=2

Please retest. Thanks.
Comment 89 psa 2010-08-22 08:29:29 UTC
I compiled the 2.6.35.3 kernel with your patch and I want to say that after the first pause for the night - everything works well:

1. make -j4  3397,29s user 260,65s system 339% cpu 17:56,23 total (i'll removed some drivers)
2. guitar pro 6 - good working
3. mplayer -vo xv - good.

However, another problem with the video card that does not wake up. But after the CM-F1 M-F7, it was restored.
Comment 90 ChriS 2010-08-22 09:25:54 UTC
It works for me.
Comment 91 psa 2010-08-23 07:28:15 UTC
Stable working after 3 suspends, good patch.
Comment 92 how 2010-08-25 12:10:07 UTC
mmm.. I don't get it. For me, it causes graphic problems like some pdf scroll really slow, some video dont play smoothly and nvidia-settings momentary freezes when starting up
Comment 93 psa 2010-08-25 13:53:29 UTC
I found another problem - after suspend I can only switch to vt7 (M-F7), yet I can't switch to any virtual terminal (1-6) - shows only illuminated black screen. Switching back to vt7 working properly.
Comment 94 Anonymous Emailer 2010-08-25 15:09:02 UTC
Reply-To: peterz@infradead.org

On Wed, 2010-08-25 at 13:53 +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> I found another problem - after suspend I can only switch to vt7 (M-F7), yet
> I
> can't switch to any virtual terminal (1-6) - shows only illuminated black
> screen. Switching back to vt7 working properly. 

Talk to the gfx people about that. I consider this bug closed.
Comment 95 how 2010-08-26 02:58:49 UTC
how is it closed when it causes other problems?
Comment 96 how 2010-09-02 02:07:53 UTC
setting maxcpus=2 with the patch solves video problems if I have 4 HT core. The patch does not entirely fixes scheduler. 

psa: you might try maxcpus=2 and see if it still have issues.
Comment 97 how 2010-09-02 12:01:48 UTC
What happens is I made a suspend/resume hook to offline one out two physical core on suspend and bring them back on on resume. This workaround works with the patch.
Comment 98 psa 2010-09-03 09:41:22 UTC
I tried maxcpus=2, but other problems, besides the above (with a switch to virtual terminals after suspend/resume) is not noticed.
Comment 99 psa 2010-10-28 07:31:41 UTC
Hi, I compiled 2.6.36 and bug present again. After suspend performance down like #82
The previous patch (fixes this bug for the 2.6.35) is present in the code of 2.6.36, but does not fix this problem.
Comment 100 how 2010-10-28 10:18:39 UTC
try this in these in your pm hooks. It works for me with the patch

offline before suspend
for AA in `seq 2 1 3` ; do echo "0" > /sys/devices/system/cpu/cpu$AA/online ; done

online after resume
for AA in `seq 2 1 3` ; do echo "1" > /sys/devices/system/cpu/cpu$AA/online ; done
Comment 101 psa 2010-10-29 07:29:50 UTC
Thanks, this hack is working.
Comment 102 how 2010-10-29 10:44:42 UTC
good stuff, you are welcome
Comment 103 how 2011-02-01 23:01:23 UTC
kernel 2.6.37 do not work with or without the hack :( why???
Comment 104 Alan 2012-06-14 17:15:04 UTC
Wondering what the current state of this is - can I close it ?
Comment 105 Florian Pritz 2012-06-14 17:17:17 UTC
My problem has been fixed, I don't know about the others. Maybe close in one or two weeks unless someone objects?
Comment 106 Alan 2012-08-29 17:24:39 UTC
thanks
Comment 107 ChriS 2012-08-29 17:35:18 UTC
I was having the same issue and I confirm that the problem is not present anymore in recent kernels.