Bug 14204

Summary: MCE prevent booting on my computer(pentium iii @500Mhz)
Product: Platform Specific/Hardware Reporter: GNUtoo (GNUtoo)
Component: i386Assignee: platform_i386
Status: CLOSED UNREPRODUCIBLE    
Severity: normal CC: andi-bz, florian, hpa, rjw
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.31 Subsystem:
Regression: Yes Bisected commit-id:
Bug Depends on:    
Bug Blocks: 13615    
Attachments: mce debug patch
dmesg

Description GNUtoo 2009-09-21 20:36:45 UTC
here's the trace obtained with serial
root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
kernel /boot/vanilla-2.6.31 root=/dev/sda3 console=ttyS0,38400
   [Linux-bzImage, setup=0x3000, size=0x370230]

[    0.000000] Linux version 2.6.31_router (root@router) (gcc version 4.1.2 (Gentoo 4.1.2 p1.3)) #1 Mon Sep 21 15:08:37 CEST 2009
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   NSC Geode by NSC
[    0.000000]   Cyrix CyrixInstead
[    0.000000]   Centaur CentaurHauls
[    0.000000]   Transmeta GenuineTMx86
[    0.000000]   Transmeta TransmetaCPU
[    0.000000]   UMC UMC UMC UMC
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (usable)
[    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 0000000013ff0000 (usable)
[    0.000000]  BIOS-e820: 0000000013ff0000 - 0000000013ff3000 (ACPI NVS)
[    0.000000]  BIOS-e820: 0000000013ff3000 - 0000000014000000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
[    0.000000] DMI 2.2 present.
[    0.000000] last_pfn = 0x13ff0 max_arch_pfn = 0x1000000
[    0.000000] PAT not supported by CPU.
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000001000 (usable)
[    0.000000]  modified: 0000000000001000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 000000000009f800 (usable)
[    0.000000]  modified: 000000000009fc00 - 00000000000a0000 (usable)
[    0.000000]  modified: 00000000000f0000 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 0000000013ff0000 (usable)
[    0.000000]  modified: 0000000013ff0000 - 0000000013ff3000 (ACPI NVS)
[    0.000000]  modified: 0000000013ff3000 - 0000000014000000 (ACPI data)
[    0.000000]  modified: 00000000ffff0000 - 0000000100000000 (reserved)
[    0.000000] init_memory_mapping: 0000000000000000-0000000013ff0000
[    0.000000] ACPI: RSDP 000f5890 00014 (v00 MITAC )
[    0.000000] ACPI: RSDT 13ff3000 00028 (v01 HP     HPCCD HW 00000002 AWRD 00000000)
[    0.000000] ACPI: FACP 13ff3040 00074 (v01 HP     HPCCD HW 00000002 AWRD 00000000)
[    0.000000] ACPI: DSDT 13ff30c0 01F12 (v01 MITAC  AWRDACPI 00001000 MSFT 0100000A)
[    0.000000] ACPI: FACS 13ff0000 00040
[    0.000000] 319MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 13ff0000
[    0.000000]   low ram: 0 - 13ff0000
[    0.000000]   node 0 low ram: 00000000 - 13ff0000
[    0.000000]   node 0 bootmap 00012000 - 00014800
[    0.000000] (6 early reservations) ==> bootmem [0000000000 - 0013ff0000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0001000000 - 00018ef2cc]    TEXT DATA BSS ==> [0001000000 - 00018ef2cc]
[    0.000000]   #2 [000009f000 - 0000100000]    BIOS reserved ==> [000009f000 - 0000100000]
[    0.000000]   #3 [00018f0000 - 00018ff0dc]              BRK ==> [00018f0000 - 00018ff0dc]
[    0.000000]   #4 [0000010000 - 0000012000]          PGTABLE ==> [0000010000 - 0000012000]
[    0.000000]   #5 [0000012000 - 0000015000]          BOOTMAP ==> [0000012000 - 0000015000]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00013ff0
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[3] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x00000001
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00013ff0
[    0.000000] Using APIC driver default
[    0.000000] ACPI: PM-Timer IO Port: 0x4008
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] Allocating PCI resources starting at 14000000 (gap: 14000000:ebff0000)
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 81152
[    0.000000] Kernel command line: root=/dev/sda3 console=ttyS0,38400
[    0.000000] PID hash table entries: 2048 (order: 11, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Enabling fast FPU save and restore... done.
[    0.000000] Enabling unmasked SIMD FPU exception support... done.
[    0.000000] Initializing CPU#0
[    0.000000] Memory: 314804k/327616k available (5976k kernel code, 12184k reserved, 2405k data, 428k init, 0k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xfffa4000 - 0xfffff000   ( 364 kB)
[    0.000000]     vmalloc : 0xd47f0000 - 0xfffa2000   ( 695 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd3ff0000   ( 319 MB)
[    0.000000]       .init : 0xc1830000 - 0xc189b000   ( 428 kB)
[    0.000000]       .data : 0xc15d620f - 0xc182f898   (2405 kB)
[    0.000000]       .text : 0xc1000000 - 0xc15d620f   (5976 kB)
[    0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.000000] SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:288
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 501.046 MHz processor.
[    0.000999] Console: colour VGA+ 80x25
[    0.000999] console [ttyS0] enabled
[    0.001022] Calibrating delay loop (skipped), value calculated using timer frequency.. 1002.09 BogoMIPS (lpj=501046)
[    0.003049] Security Framework initialized
[    0.004035] SELinux:  Initializing.
[    0.005083] Mount-cache hash table entries: 512
[    0.007522] CPU: L1 I cache: 16K, L1 D cache: 16K
[    0.009008] CPU: L2 cache: 512K
[    0.010016] mce: CPU supports 5 MCE banks
[    0.011029] general protection fault: 0000 [#1] 
[    0.011998] last sysfs file: 
[    0.011998] Modules linked in:
[    0.011998] 
[    0.011998] Pid: 0, comm: swapper Not tainted (2.6.31_router #1) HP Vectra
[    0.011998] EIP: 0060:[<c100d9b9>] EFLAGS: 00010246 CPU: 0
[    0.011998] EIP is at mce_rdmsrl+0x19/0x60
[    0.011998] EAX: 00000000 EBX: 00000001 ECX: 00000407 EDX: 08000000
[    0.011998] ESI: 00000000 EDI: 8c000000 EBP: 00000405 ESP: c17d5eac
[    0.011998]  DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068
[    0.011998] Process swapper (pid: 0, ti=c17d4000 task=c17db300 task.ti=c17d4000)
[    0.011998] Stack:
[    0.011998]  860f46b5 c100f0ac c17d5f48 00000006 0000000f 860f46b5 00000000 8c000000
[    0.011998] <0> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    0.011998] <0> 00000000 00000000 4ab7b906 00000000 00000000 00000673 00000100 00000000
[    0.011998] Call Trace:
[    0.011998]  [<c100f0ac>] ? machine_check_poll+0x1bc/0x1d0
[    0.011998]  [<c100e720>] ? do_machine_check+0x0/0x7d0
[    0.011998]  [<c100f101>] ? mce_init+0x41/0x100
[    0.011998]  [<c186d04e>] ? mcheck_init+0x17e/0x3c0
[    0.011998]  [<c186b9a9>] ? init_intel+0x149/0x400
[    0.011998]  [<c186ad55>] ? get_cpu_vendor+0x25/0xa0
[    0.011998]  [<c186afd8>] ? identify_cpu+0x198/0x380
[    0.011998]  [<c1830360>] ? unknown_bootoption+0x0/0x210
[    0.011998]  [<c18379ea>] ? identify_boot_cpu+0xa/0x20
[    0.011998]  [<c1837b24>] ? check_bugs+0x14/0x110
[    0.011998]  [<c1830a15>] ? start_kernel+0x275/0x310
[    0.011998]  [<c1830360>] ? unknown_bootoption+0x0/0x210
[    0.011998]  [<c1830055>] ? __init_begin+0x55/0xa0
[    0.011998] Code: da e8 5c 06 02 00 8d b6 00 00 00 00 8d bf 00 00 00 00 83 ec 04 89 c1 65 a1 14 00 00 00 89 04 24 31 c0 80 3d e3 ff 7d c1 00 75 17 <0f> 32 8b 0c 24 65 33 0d 14 00 00 00 75 28 59 c3 8d b4 26 00 00 
[    0.011998] EIP: [<c100d9b9>] mce_rdmsrl+0x19/0x60 SS:ESP 0068:c17d5eac
[    0.012034] ---[ end trace a7919e7f17c0a725 ]---
[    0.013011] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.014013] Pid: 0, comm: swapper Tainted: G      D    2.6.31_router #1
[    0.015006] Call Trace:
[    0.016019]  [<c102df45>] ? panic+0x55/0x100
[    0.017020]  [<c1030981>] ? do_exit+0x571/0x610
[    0.018022]  [<c1004040>] ? do_general_protection+0x0/0x1e0
[    0.019015]  [<c102eee7>] ? printk+0x27/0x50
[    0.020017]  [<c1004040>] ? do_general_protection+0x0/0x1e0
[    0.022014]  [<c1006621>] ? oops_end+0xa1/0xb0
[    0.023016]  [<c15d55fb>] ? error_code+0x6b/0x70
[    0.024018]  [<c131007b>] ? pps_poweroff_atca+0xb/0xa0
[    0.025015]  [<c100d9b9>] ? mce_rdmsrl+0x19/0x60
[    0.026016]  [<c100f0ac>] ? machine_check_poll+0x1bc/0x1d0
[    0.027018]  [<c100e720>] ? do_machine_check+0x0/0x7d0
[    0.028014]  [<c100f101>] ? mce_init+0x41/0x100
[    0.029016]  [<c186d04e>] ? mcheck_init+0x17e/0x3c0
[    0.030018]  [<c186b9a9>] ? init_intel+0x149/0x400
[    0.031015]  [<c186ad55>] ? get_cpu_vendor+0x25/0xa0
[    0.032016]  [<c186afd8>] ? identify_cpu+0x198/0x380
[    0.033017]  [<c1830360>] ? unknown_bootoption+0x0/0x210
[    0.034014]  [<c18379ea>] ? identify_boot_cpu+0xa/0x20
[    0.035012]  [<c1837b24>] ? check_bugs+0x14/0x110
[    0.036014]  [<c1830a15>] ? start_kernel+0x275/0x310
[    0.037013]  [<c1830360>] ? unknown_bootoption+0x0/0x210
[    0.038013]  [<c1830055>] ? __init_begin+0x55/0xa0
Comment 1 GNUtoo 2009-09-21 20:37:14 UTC
disabling all mce options make it boot...
Comment 2 Andi Kleen 2009-09-23 15:24:42 UTC
Need more information.

What kind of hardware is that. Please post /proc/cpuinfo

Did 2.6.30/32bit or a earlier kernel 
with MCE enabled work?  Please attach the boot log from such a successfull 
boot.

Thanks.
Comment 3 Ingo Molnar 2009-09-23 16:17:21 UTC
Thanks for your report. Could you please try the latest -tip tree:

  http://people.redhat.com/mingo/tip.git/README

With the MCE options enabled? It should not crash anymore due to this 
robustness fix:

  11868a2: x86: mce: Use safer ways to access MCE registers

If the patch works as intended then you should be getting a warning 
message during bootup - please post that bootlog.

Thanks,

	Ingo
Comment 4 Andi Kleen 2009-09-23 18:02:12 UTC
Ingo, as I explained several times rdmsrl_safe() is very unlikely to the 
correct fix. If the MSR code reads MSRs that are not there something
is going wrong in the bank or feature discovery and we need to root 
cause what it is, not hack around it at the wrong level. 

I think if you spend 5 minutes reading the respective chapters
in the architecture manual you will get to the same conclusion.

The fault seems to happen on MSR 407 which is the MISC register in bank 2.
Bank 2 should be really there, but it might have no MISC register,
which is optional (iirc P3 really didn't have a MISC register)

The MISC register normally is specified by setting the MISCV bit,
which might be set incorrectly here. 

BTW I think this can also only happen if there is a machine check
logged at time of boot.  If it happens on every boot (does it?)
then most likely the BIOS leaves junk in the machine check registers,
including one with a MISCV bit, but MISC isn't there.

So in this case the correct fix would be to add that system to the
"don't log boot mces" blacklist. To add it I would need the cpuinfo 
output.

It would be also interesting to see what the status register say.
Comment 5 Andi Kleen 2009-09-23 18:06:40 UTC
Created attachment 23153 [details]
mce debug patch

Here's a debug patch that prints all the registers on a MISC dump.
Please apply it to 2.6.31 (reenabling mces in the config) and add the boot
log. It should not crash. Thanks.

Also another workaround would be to boot with mce=nobootlog
Comment 6 Ingo Molnar 2009-09-23 18:57:50 UTC
> Ingo, as I explained several times rdmsrl_safe() is very unlikely to 
> the correct fix. [...]

Andi, as i told you before, this is a robustness improvement: a 'turn 
nasty boot crash into a debuggable boot warning' patch.

It does not fix this bug, obviously, nor does it claim to.

It is a robustness fix that you failed to add and which even today you 
fail to realize the significance of.

There's a world of a difference between a boot crash and a boot warning. 
Boot warnings can be debugged and reported much more easily than boot 
crashes. Who knows how many boxes crashed silently due to the bug which 
people were unable to report in a meaningful way.

	Ingo
Comment 7 GNUtoo 2009-09-23 19:02:02 UTC
here are the requested informations...sorry for not beeing able to respond at once
router ~ # cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 7
model name	: Pentium III (Katmai)
stepping	: 3
cpu MHz		: 501.167
cache size	: 512 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips	: 1002.33
clflush size	: 32
power management:

and here's my last kernel(vanilla 2.6.29)
router ~ # uname -a
Linux router 2.6.29_router #1 Wed Mar 25 15:47:10 CET 2009 i686 Pentium III (Katmai) GenuineIntel GNU/Linux
Comment 8 Andi Kleen 2009-09-23 23:33:55 UTC
Re #6: Ingo, ok if you don't treat it as the final fix for this I have no objections. It just didn't sound like that from your first comment, but I might have misread. The important part is that it gets reported, but the WARN_ON
will insure that.

Re #7:
Gnutoo, Can you please apply the patch I attached to this bug to 2.6.31, reenable machine  checks and attach the resulting boot log after boot? Thanks.
Comment 9 GNUtoo 2009-09-24 09:47:03 UTC
this time it booted with the patch:
root@router ~ # zcat /proc/config.gz |  grep MCE
zcat /proc/config.gz |  grep MCE
CONFIG_X86_MCE=y
# CONFIG_X86_OLD_MCE is not set
CONFIG_X86_NEW_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
# CONFIG_X86_ANCIENT_MCE is not set
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set

I'll attach the dmesg
Comment 10 GNUtoo 2009-09-24 09:47:59 UTC
Created attachment 23169 [details]
dmesg
Comment 11 Ingo Molnar 2009-09-24 11:21:09 UTC
> Re #6: Ingo, ok if you don't treat it as the final fix for this I have 
> no objections. It just didn't sound like that from your first comment, 
> but I might have misread. The important part is that it gets reported, 
> but the WARN_ON will insure that.

Andi, you either have serious reading comprehension problems, or you 
pretend that you didnt get the point i made. To quote the very plain 
language of the commit:

------------>
  commit 11868a2dc4f5e4f2f652bfd259e1360193fcee62
  Author: Ingo Molnar <mingo@elte.hu>
  Date:   Wed Sep 23 17:49:55 2009 +0200

    x86: mce: Use safer ways to access MCE registers
    [...]

    So WARN_ONCE() instead of crashing the box.
<------------

	Ingo
Comment 12 Ingo Molnar 2009-09-24 11:39:40 UTC
GNUtoo: the latest dmesg you sent with the debug patch applied does not 
have an MCE event printed in it. That suggests sensitivity of this MCE 
event on the specific kernel layout - or on other random factors.

What is the output of /proc/interrupts after bootup? It should contain 
an 'MCE' line like this:

 MCE:          0   Machine check exceptions

It will be non-zero if you got MCE exceptions.

Does this system have any history of thermal instability?

Also, could you check the latest -tip tree at:

   http://people.redhat.com/mingo/tip.git/README

and attach the dmesg with that booted. Do you get an MCE exception with 
that, and does the new WARN() trigger?

Thanks,

	Ingo
Comment 13 GNUtoo 2009-10-01 17:23:18 UTC
router ~ # cat /proc/interrupts 
           CPU0       
  0:      58287    XT-PIC-XT        timer
  1:          2    XT-PIC-XT        i8042
  2:          0    XT-PIC-XT        cascade
  3:          1    XT-PIC-XT        uhci_hcd:usb1
  4:        510    XT-PIC-XT        serial
  7:          2    XT-PIC-XT      
  9:          0    XT-PIC-XT        acpi
 10:          1    XT-PIC-XT        eth0
 11:        810    XT-PIC-XT        b43
 12:         67    XT-PIC-XT        CS46XX, eth1
 14:       2930    XT-PIC-XT        ata_piix
 15:          0    XT-PIC-XT        ata_piix
NMI:          0   Non-mas   Local timer interrupts
SPU:          0   Spurious interrupts
CNT:          0   Performance counter interrupts
PND:          0   Performance pending work
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          

maybe it was because it just compiled the kernel and was too hot
Comment 14 H. Peter Anvin 2009-10-01 17:38:44 UTC
Well, that could easily have triggered a machine check event, that if it was mishandled, would have crashed the machine...
Comment 15 Florian Mickler 2010-12-17 21:09:35 UTC
I'm resolving this ancient issue as "unreproducible". Might have been too hot or might have been cosmic rays. Please shout if this is incorrect. 

(I'm not shure if you have to shout, if you can reproduce this issue, but only while putting your computer in the oven. In that case it might be theoretically be more correct to resolve this issue as "invalid".. but I'm not shure we care that deeply.)