here's the trace obtained with serial root (hd0,0) Filesystem type is ext2fs, partition type 0x83 kernel /boot/vanilla-2.6.31 root=/dev/sda3 console=ttyS0,38400 [Linux-bzImage, setup=0x3000, size=0x370230] [ 0.000000] Linux version 2.6.31_router (root@router) (gcc version 4.1.2 (Gentoo 4.1.2 p1.3)) #1 Mon Sep 21 15:08:37 CEST 2009 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] NSC Geode by NSC [ 0.000000] Cyrix CyrixInstead [ 0.000000] Centaur CentaurHauls [ 0.000000] Transmeta GenuineTMx86 [ 0.000000] Transmeta TransmetaCPU [ 0.000000] UMC UMC UMC UMC [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f800 (usable) [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (usable) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 0000000013ff0000 (usable) [ 0.000000] BIOS-e820: 0000000013ff0000 - 0000000013ff3000 (ACPI NVS) [ 0.000000] BIOS-e820: 0000000013ff3000 - 0000000014000000 (ACPI data) [ 0.000000] BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved) [ 0.000000] DMI 2.2 present. [ 0.000000] last_pfn = 0x13ff0 max_arch_pfn = 0x1000000 [ 0.000000] PAT not supported by CPU. [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] modified physical RAM map: [ 0.000000] modified: 0000000000000000 - 0000000000001000 (usable) [ 0.000000] modified: 0000000000001000 - 0000000000010000 (reserved) [ 0.000000] modified: 0000000000010000 - 000000000009f800 (usable) [ 0.000000] modified: 000000000009fc00 - 00000000000a0000 (usable) [ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] modified: 0000000000100000 - 0000000013ff0000 (usable) [ 0.000000] modified: 0000000013ff0000 - 0000000013ff3000 (ACPI NVS) [ 0.000000] modified: 0000000013ff3000 - 0000000014000000 (ACPI data) [ 0.000000] modified: 00000000ffff0000 - 0000000100000000 (reserved) [ 0.000000] init_memory_mapping: 0000000000000000-0000000013ff0000 [ 0.000000] ACPI: RSDP 000f5890 00014 (v00 MITAC ) [ 0.000000] ACPI: RSDT 13ff3000 00028 (v01 HP HPCCD HW 00000002 AWRD 00000000) [ 0.000000] ACPI: FACP 13ff3040 00074 (v01 HP HPCCD HW 00000002 AWRD 00000000) [ 0.000000] ACPI: DSDT 13ff30c0 01F12 (v01 MITAC AWRDACPI 00001000 MSFT 0100000A) [ 0.000000] ACPI: FACS 13ff0000 00040 [ 0.000000] 319MB LOWMEM available. [ 0.000000] mapped low ram: 0 - 13ff0000 [ 0.000000] low ram: 0 - 13ff0000 [ 0.000000] node 0 low ram: 00000000 - 13ff0000 [ 0.000000] node 0 bootmap 00012000 - 00014800 [ 0.000000] (6 early reservations) ==> bootmem [0000000000 - 0013ff0000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] [ 0.000000] #1 [0001000000 - 00018ef2cc] TEXT DATA BSS ==> [0001000000 - 00018ef2cc] [ 0.000000] #2 [000009f000 - 0000100000] BIOS reserved ==> [000009f000 - 0000100000] [ 0.000000] #3 [00018f0000 - 00018ff0dc] BRK ==> [00018f0000 - 00018ff0dc] [ 0.000000] #4 [0000010000 - 0000012000] PGTABLE ==> [0000010000 - 0000012000] [ 0.000000] #5 [0000012000 - 0000015000] BOOTMAP ==> [0000012000 - 0000015000] [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] Normal 0x00001000 -> 0x00013ff0 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[3] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x00000001 [ 0.000000] 0: 0x00000010 -> 0x0000009f [ 0.000000] 0: 0x00000100 -> 0x00013ff0 [ 0.000000] Using APIC driver default [ 0.000000] ACPI: PM-Timer IO Port: 0x4008 [ 0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic" [ 0.000000] APIC: disable apic facility [ 0.000000] Allocating PCI resources starting at 14000000 (gap: 14000000:ebff0000) [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 81152 [ 0.000000] Kernel command line: root=/dev/sda3 console=ttyS0,38400 [ 0.000000] PID hash table entries: 2048 (order: 11, 8192 bytes) [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) [ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) [ 0.000000] Enabling fast FPU save and restore... done. [ 0.000000] Enabling unmasked SIMD FPU exception support... done. [ 0.000000] Initializing CPU#0 [ 0.000000] Memory: 314804k/327616k available (5976k kernel code, 12184k reserved, 2405k data, 428k init, 0k highmem) [ 0.000000] virtual kernel memory layout: [ 0.000000] fixmap : 0xfffa4000 - 0xfffff000 ( 364 kB) [ 0.000000] vmalloc : 0xd47f0000 - 0xfffa2000 ( 695 MB) [ 0.000000] lowmem : 0xc0000000 - 0xd3ff0000 ( 319 MB) [ 0.000000] .init : 0xc1830000 - 0xc189b000 ( 428 kB) [ 0.000000] .data : 0xc15d620f - 0xc182f898 (2405 kB) [ 0.000000] .text : 0xc1000000 - 0xc15d620f (5976 kB) [ 0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok. [ 0.000000] SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:288 [ 0.000000] Fast TSC calibration using PIT [ 0.000000] Detected 501.046 MHz processor. [ 0.000999] Console: colour VGA+ 80x25 [ 0.000999] console [ttyS0] enabled [ 0.001022] Calibrating delay loop (skipped), value calculated using timer frequency.. 1002.09 BogoMIPS (lpj=501046) [ 0.003049] Security Framework initialized [ 0.004035] SELinux: Initializing. [ 0.005083] Mount-cache hash table entries: 512 [ 0.007522] CPU: L1 I cache: 16K, L1 D cache: 16K [ 0.009008] CPU: L2 cache: 512K [ 0.010016] mce: CPU supports 5 MCE banks [ 0.011029] general protection fault: 0000 [#1] [ 0.011998] last sysfs file: [ 0.011998] Modules linked in: [ 0.011998] [ 0.011998] Pid: 0, comm: swapper Not tainted (2.6.31_router #1) HP Vectra [ 0.011998] EIP: 0060:[<c100d9b9>] EFLAGS: 00010246 CPU: 0 [ 0.011998] EIP is at mce_rdmsrl+0x19/0x60 [ 0.011998] EAX: 00000000 EBX: 00000001 ECX: 00000407 EDX: 08000000 [ 0.011998] ESI: 00000000 EDI: 8c000000 EBP: 00000405 ESP: c17d5eac [ 0.011998] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 [ 0.011998] Process swapper (pid: 0, ti=c17d4000 task=c17db300 task.ti=c17d4000) [ 0.011998] Stack: [ 0.011998] 860f46b5 c100f0ac c17d5f48 00000006 0000000f 860f46b5 00000000 8c000000 [ 0.011998] <0> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 0.011998] <0> 00000000 00000000 4ab7b906 00000000 00000000 00000673 00000100 00000000 [ 0.011998] Call Trace: [ 0.011998] [<c100f0ac>] ? machine_check_poll+0x1bc/0x1d0 [ 0.011998] [<c100e720>] ? do_machine_check+0x0/0x7d0 [ 0.011998] [<c100f101>] ? mce_init+0x41/0x100 [ 0.011998] [<c186d04e>] ? mcheck_init+0x17e/0x3c0 [ 0.011998] [<c186b9a9>] ? init_intel+0x149/0x400 [ 0.011998] [<c186ad55>] ? get_cpu_vendor+0x25/0xa0 [ 0.011998] [<c186afd8>] ? identify_cpu+0x198/0x380 [ 0.011998] [<c1830360>] ? unknown_bootoption+0x0/0x210 [ 0.011998] [<c18379ea>] ? identify_boot_cpu+0xa/0x20 [ 0.011998] [<c1837b24>] ? check_bugs+0x14/0x110 [ 0.011998] [<c1830a15>] ? start_kernel+0x275/0x310 [ 0.011998] [<c1830360>] ? unknown_bootoption+0x0/0x210 [ 0.011998] [<c1830055>] ? __init_begin+0x55/0xa0 [ 0.011998] Code: da e8 5c 06 02 00 8d b6 00 00 00 00 8d bf 00 00 00 00 83 ec 04 89 c1 65 a1 14 00 00 00 89 04 24 31 c0 80 3d e3 ff 7d c1 00 75 17 <0f> 32 8b 0c 24 65 33 0d 14 00 00 00 75 28 59 c3 8d b4 26 00 00 [ 0.011998] EIP: [<c100d9b9>] mce_rdmsrl+0x19/0x60 SS:ESP 0068:c17d5eac [ 0.012034] ---[ end trace a7919e7f17c0a725 ]--- [ 0.013011] Kernel panic - not syncing: Attempted to kill the idle task! [ 0.014013] Pid: 0, comm: swapper Tainted: G D 2.6.31_router #1 [ 0.015006] Call Trace: [ 0.016019] [<c102df45>] ? panic+0x55/0x100 [ 0.017020] [<c1030981>] ? do_exit+0x571/0x610 [ 0.018022] [<c1004040>] ? do_general_protection+0x0/0x1e0 [ 0.019015] [<c102eee7>] ? printk+0x27/0x50 [ 0.020017] [<c1004040>] ? do_general_protection+0x0/0x1e0 [ 0.022014] [<c1006621>] ? oops_end+0xa1/0xb0 [ 0.023016] [<c15d55fb>] ? error_code+0x6b/0x70 [ 0.024018] [<c131007b>] ? pps_poweroff_atca+0xb/0xa0 [ 0.025015] [<c100d9b9>] ? mce_rdmsrl+0x19/0x60 [ 0.026016] [<c100f0ac>] ? machine_check_poll+0x1bc/0x1d0 [ 0.027018] [<c100e720>] ? do_machine_check+0x0/0x7d0 [ 0.028014] [<c100f101>] ? mce_init+0x41/0x100 [ 0.029016] [<c186d04e>] ? mcheck_init+0x17e/0x3c0 [ 0.030018] [<c186b9a9>] ? init_intel+0x149/0x400 [ 0.031015] [<c186ad55>] ? get_cpu_vendor+0x25/0xa0 [ 0.032016] [<c186afd8>] ? identify_cpu+0x198/0x380 [ 0.033017] [<c1830360>] ? unknown_bootoption+0x0/0x210 [ 0.034014] [<c18379ea>] ? identify_boot_cpu+0xa/0x20 [ 0.035012] [<c1837b24>] ? check_bugs+0x14/0x110 [ 0.036014] [<c1830a15>] ? start_kernel+0x275/0x310 [ 0.037013] [<c1830360>] ? unknown_bootoption+0x0/0x210 [ 0.038013] [<c1830055>] ? __init_begin+0x55/0xa0
disabling all mce options make it boot...
Need more information. What kind of hardware is that. Please post /proc/cpuinfo Did 2.6.30/32bit or a earlier kernel with MCE enabled work? Please attach the boot log from such a successfull boot. Thanks.
Thanks for your report. Could you please try the latest -tip tree: http://people.redhat.com/mingo/tip.git/README With the MCE options enabled? It should not crash anymore due to this robustness fix: 11868a2: x86: mce: Use safer ways to access MCE registers If the patch works as intended then you should be getting a warning message during bootup - please post that bootlog. Thanks, Ingo
Ingo, as I explained several times rdmsrl_safe() is very unlikely to the correct fix. If the MSR code reads MSRs that are not there something is going wrong in the bank or feature discovery and we need to root cause what it is, not hack around it at the wrong level. I think if you spend 5 minutes reading the respective chapters in the architecture manual you will get to the same conclusion. The fault seems to happen on MSR 407 which is the MISC register in bank 2. Bank 2 should be really there, but it might have no MISC register, which is optional (iirc P3 really didn't have a MISC register) The MISC register normally is specified by setting the MISCV bit, which might be set incorrectly here. BTW I think this can also only happen if there is a machine check logged at time of boot. If it happens on every boot (does it?) then most likely the BIOS leaves junk in the machine check registers, including one with a MISCV bit, but MISC isn't there. So in this case the correct fix would be to add that system to the "don't log boot mces" blacklist. To add it I would need the cpuinfo output. It would be also interesting to see what the status register say.
Created attachment 23153 [details] mce debug patch Here's a debug patch that prints all the registers on a MISC dump. Please apply it to 2.6.31 (reenabling mces in the config) and add the boot log. It should not crash. Thanks. Also another workaround would be to boot with mce=nobootlog
> Ingo, as I explained several times rdmsrl_safe() is very unlikely to > the correct fix. [...] Andi, as i told you before, this is a robustness improvement: a 'turn nasty boot crash into a debuggable boot warning' patch. It does not fix this bug, obviously, nor does it claim to. It is a robustness fix that you failed to add and which even today you fail to realize the significance of. There's a world of a difference between a boot crash and a boot warning. Boot warnings can be debugged and reported much more easily than boot crashes. Who knows how many boxes crashed silently due to the bug which people were unable to report in a meaningful way. Ingo
here are the requested informations...sorry for not beeing able to respond at once router ~ # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 7 model name : Pentium III (Katmai) stepping : 3 cpu MHz : 501.167 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1002.33 clflush size : 32 power management: and here's my last kernel(vanilla 2.6.29) router ~ # uname -a Linux router 2.6.29_router #1 Wed Mar 25 15:47:10 CET 2009 i686 Pentium III (Katmai) GenuineIntel GNU/Linux
Re #6: Ingo, ok if you don't treat it as the final fix for this I have no objections. It just didn't sound like that from your first comment, but I might have misread. The important part is that it gets reported, but the WARN_ON will insure that. Re #7: Gnutoo, Can you please apply the patch I attached to this bug to 2.6.31, reenable machine checks and attach the resulting boot log after boot? Thanks.
this time it booted with the patch: root@router ~ # zcat /proc/config.gz | grep MCE zcat /proc/config.gz | grep MCE CONFIG_X86_MCE=y # CONFIG_X86_OLD_MCE is not set CONFIG_X86_NEW_MCE=y CONFIG_X86_MCE_INTEL=y CONFIG_X86_MCE_AMD=y # CONFIG_X86_ANCIENT_MCE is not set CONFIG_X86_MCE_THRESHOLD=y # CONFIG_X86_MCE_INJECT is not set I'll attach the dmesg
Created attachment 23169 [details] dmesg
> Re #6: Ingo, ok if you don't treat it as the final fix for this I have > no objections. It just didn't sound like that from your first comment, > but I might have misread. The important part is that it gets reported, > but the WARN_ON will insure that. Andi, you either have serious reading comprehension problems, or you pretend that you didnt get the point i made. To quote the very plain language of the commit: ------------> commit 11868a2dc4f5e4f2f652bfd259e1360193fcee62 Author: Ingo Molnar <mingo@elte.hu> Date: Wed Sep 23 17:49:55 2009 +0200 x86: mce: Use safer ways to access MCE registers [...] So WARN_ONCE() instead of crashing the box. <------------ Ingo
GNUtoo: the latest dmesg you sent with the debug patch applied does not have an MCE event printed in it. That suggests sensitivity of this MCE event on the specific kernel layout - or on other random factors. What is the output of /proc/interrupts after bootup? It should contain an 'MCE' line like this: MCE: 0 Machine check exceptions It will be non-zero if you got MCE exceptions. Does this system have any history of thermal instability? Also, could you check the latest -tip tree at: http://people.redhat.com/mingo/tip.git/README and attach the dmesg with that booted. Do you get an MCE exception with that, and does the new WARN() trigger? Thanks, Ingo
router ~ # cat /proc/interrupts CPU0 0: 58287 XT-PIC-XT timer 1: 2 XT-PIC-XT i8042 2: 0 XT-PIC-XT cascade 3: 1 XT-PIC-XT uhci_hcd:usb1 4: 510 XT-PIC-XT serial 7: 2 XT-PIC-XT 9: 0 XT-PIC-XT acpi 10: 1 XT-PIC-XT eth0 11: 810 XT-PIC-XT b43 12: 67 XT-PIC-XT CS46XX, eth1 14: 2930 XT-PIC-XT ata_piix 15: 0 XT-PIC-XT ata_piix NMI: 0 Non-mas Local timer interrupts SPU: 0 Spurious interrupts CNT: 0 Performance counter interrupts PND: 0 Performance pending work TRM: 0 Thermal event interrupts THR: 0 Threshold APIC interrupts MCE: maybe it was because it just compiled the kernel and was too hot
Well, that could easily have triggered a machine check event, that if it was mishandled, would have crashed the machine...
I'm resolving this ancient issue as "unreproducible". Might have been too hot or might have been cosmic rays. Please shout if this is incorrect. (I'm not shure if you have to shout, if you can reproduce this issue, but only while putting your computer in the oven. In that case it might be theoretically be more correct to resolve this issue as "invalid".. but I'm not shure we care that deeply.)