Bug 8346 - boot crash + hangduring modprobe processor unless "processor.nocst" - P4/HT
Summary: boot crash + hangduring modprobe processor unless "processor.nocst" - P4/HT
Status: CLOSED CODE_FIX
Alias: None
Product: ACPI
Classification: Unclassified
Component: BIOS (show other bugs)
Hardware: i386 Linux
: P2 high
Assignee: Len Brown
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-04-17 01:01 UTC by Olaf Kirch
Modified: 2007-04-30 20:38 UTC (History)
1 user (show)

See Also:
Kernel Version: v2.6.21-rc7
Tree: Mainline
Regression: ---


Attachments
acpidump (77.77 KB, text/plain)
2007-04-17 05:08 UTC, Olaf Kirch
Details
dmidecode (19.60 KB, text/plain)
2007-04-17 05:08 UTC, Olaf Kirch
Details
Possible fix (1.01 KB, patch)
2007-04-19 00:53 UTC, Olaf Kirch
Details | Diff
Snapshot of an oops (162.06 KB, image/jpeg)
2007-04-19 04:29 UTC, Olaf Kirch
Details
Another oops (207.39 KB, image/jpeg)
2007-04-19 04:40 UTC, Olaf Kirch
Details
patch vs 2.6.21-rc7 (1.33 KB, patch)
2007-04-25 21:29 UTC, Len Brown
Details | Diff
patch for 2.6.22-git1 (1.66 KB, patch)
2007-04-28 18:01 UTC, Len Brown
Details | Diff

Description Olaf Kirch 2007-04-17 01:01:41 UTC
Most recent kernel where this bug did *NOT* occur:v2.6.20-rc7
Distribution: Suse user land
Hardware Environment: P4
Software Environment:
Problem Description:
In roughly 2 out of 4 boots, the machine will crash or freeze during boot
while loading processor.ko from initrd. It frequently just hangs, no kbd,
no sysrq or anything. Sometimes it produces oopses, but it doesn't always
oops in the same place. Sometimes I get a spontaneous reboot; at least once
I had something trash the frame buffer with a stipple pattern.

I was unable to reproduce the bug without processor.ko.

acpi_serialize doesn't make a difference.

Common characteristics of all backtraces as far as I was able to gather them
	acpi_processor_init
	-> acpi_processor_power_init
	-> (some ACPI eval related stuff)
	-> do_page_fault
	-> fixup_exception
	-> search_exception_table
	-> _spin_lock_irqsave

Sometimes, the EIP is bad (eg 0x246)

In one oops I captured, segment registers were hosed, with
   ds:0920 es:a574 fs:3834 gs:0033 ss:0068

So, this looks like memory corruption. I'm currently working on a bisect.

Steps to reproduce:
Comment 1 Olaf Kirch 2007-04-17 05:08:06 UTC
Created attachment 11217 [details]
acpidump
Comment 2 Olaf Kirch 2007-04-17 05:08:44 UTC
Created attachment 11218 [details]
dmidecode
Comment 3 Olaf Kirch 2007-04-17 05:32:47 UTC
I narrowed it down a little. The problem was introduced by the
ACPICA merge

good: e47fddf2470feb228e1d3ff41fc78dad4cfbbcc6
bad:  15a58ed12142939d51076380e6e58af477ad96ec

I'll try to drill into this a little more, but unfortunately a lot of
changes in this set of patches don't compile, which makes bisect a little
tedious
Comment 4 Olaf Kirch 2007-04-17 07:32:10 UTC
It seems to boil down to one of these:

2502fffb1958da66fa50a475081cb6827acdd9f3 ACPICA: Add support for DMAR table
ad71860a17ba33eb0e673e9e2cf5ba0d8e3e3fdd ACPICA: minimal patch to integrate new 
tables into Linux
a4bbb810dedaecf74d54b16b6dd3c33e95e1024c ACPICA: Lint changes
4bf273939c99fae5bae399f51c417a552d74b97f ACPICA: Fix for FADT conversion in 
64-bit mode
8f34890dce60f7df6dd23a0d04977c6572adaab8 ACPICA: Update comments for individual 
table fields
c5fc42ac4d4d6d3e3f619290b86890cb3725d2f8 ACPICA: misc fixes for new Table 
Manager:
f3d2e7865c816258c699ff965768e46b50d536d3 ACPICA: Implement simplified Table 
Manager

Unfortunately, these are rather big. And I'm unable to bisect further, I
keep getting compile errors in misc places (mostly es7000 code, but elsewhere
too - eg
In file included from arch/i386/kernel/setup.c:31:
include/linux/acpi.h:59: error: redefinition of ‘struct acpi_table_rsdp’
include/linux/acpi.h:86: error: redefinition of ‘struct acpi_table_rsdt’
include/linux/acpi.h:93: error: redefinition of ‘struct acpi_table_xsdt’
include/linux/acpi.h:100: error: redefinition of ‘struct acpi_table_fadt’
Comment 5 Olaf Kirch 2007-04-17 12:17:26 UTC
I enabled ACPI debugging and noticed that the BIOS doesn't have a _CST.
However, the following if() is executed:

if (acpi_fadt.cst_control && !nocst) {
        status =
		acpi_os_write_port(acpi_fadt.smi_command,
			acpi_fadt.cst_control, 8);
                [...]
}

If I set nocst, the boot seems to proceed normally. cst_control has a value
of 0xe3.
Comment 6 Len Brown 2007-04-18 18:57:54 UTC
So the problem goes away if you boot with "processor.nocst"?
Comment 7 Olaf Kirch 2007-04-19 00:41:34 UTC
Yes. FWIW, if I just skip writing cst_control if no _CST was found,
this will work as well. The patch below will call acpi_processor_get_power_info
first, and only write CST_CNT to the SMI_CMND register iff pr->flags.has_cst
is true.
Comment 8 Olaf Kirch 2007-04-19 00:53:27 UTC
Created attachment 11229 [details]
Possible fix

I'm not entirely sure if this is the right approach. I understand that
the _CST object is supposed to be dynamic, but can the BIOS hide it
completely prior to the OSPM sending CST_CNT?
Comment 9 Olaf Kirch 2007-04-19 04:29:34 UTC
Created attachment 11230 [details]
Snapshot of an oops

The longer I look at this, the more I suspect something else is
going wrong here. Look at the oops - it's strange in many ways.

 -	It's the swapper task, but the stack looks like modprobe's
	There's a mismatch between ti and task.ti - current points
	at the swapper task_struct but %esp is that of the modprobe
	task

 -	The EIP shown is bogus: 7E70:[0000001e] is crap. 0x1e matches %edi,
	and 7e70 is the lower 16bit of %ebp

 -	ds and es segment registers are crap too; these correspond to the
	topmost two words on the stack shown

Note this is oops #2, the first one had scrolled off the top of the
screen and scrollback was not possible.
Comment 10 Olaf Kirch 2007-04-19 04:40:28 UTC
Created attachment 11231 [details]
Another oops

Here's another oops, with similar characteristics

Again it's the swapper task, the pt_regs shown look rather corrupted,
and the stack shown is modprobe's.

The most peculiar thing about this oops is EIP.
It says:

printing eip
c16a9080
...
EIP:  53b8:[<c0243b8>]
...
Code: 00 00 00 00
EIP: [<00000000>]

So it shows three different EIP values in one single oops. It appears as if
the stack is still active while being printed. Which isn't that surprising
given that we're dumping the stack of another task, which may still be
running on the other sibling CPU.
Comment 11 Olaf Kirch 2007-04-20 07:16:23 UTC
This bug is a bit frustrating :-)

I spent another day on trying to hunt down exactly what is happening,
with moderate success. Here's what I could establish for a fact

 -	the bug is triggered by the call to
	acpi_os_write_port(acpi_gbl_FADT.smi_command,
		acpi_gbl_FADT.cst_control, 8);
	If I make it skip this outb, everything works like a charm

 -	The symptoms look like some strange memory corruption to me, but
	without a fixed pattern. However, many of the oopses I recorded
	had a mismatch between %esp (which belonged to modprobe) and
	struct current (which pointed at some other task, probably whatever
	was currently active on the other CPU).
	I actually added a check to the oops code which would chase
	current_thread_info()->task, and it would always show the modprobe
	task in state TASK_RUNNING.

 -	In the oops handler, I compared smp_processor_id() to
	safe_smp_processor_id(), which I *think* should trigger if we're
	running with %fs pointing to the wrong kernel PDA. But this condition
	never triggered.

 -	I added a printk to acpi_ev_sci_xrupt_handler and a few exception
	handlers to see if there was some bad interaction going on, but
	that printk never triggered.

So I'm back to suspecting the ACPI BIOS doing something stupid - which would
be a pretty good assumption if it wasn't for the fact that the git bisect
showed that this particular problem seems to have been introduced by the
new ACPI table manager code.

Any other ideas to try?
Comment 12 Len Brown 2007-04-25 20:12:15 UTC
This system has a revision 2 FADT,
but it is populating FADT.CST_CONTROL.

However, the FADT.CST_CONTROL field was RESERVED until
FADT revision 3 and ACPI 2.0 -- so a non-zero value
here is technically a BIOS bug.

Linux is believing the BIOS that it should send these
bits to the SMI_COMMAND register in order to tell
the BIOS that Linux knows how to handle a _CST.

As this system doesn't have a CST, it is technically
a 2nd BIOS bug that this field is populated -- even
if the BIOS were up to date with ACPI 2.0.

Which brings us to the actual (intermittant) failure.
We've had problems in the past with tickling SMM
from a thread other than CPU0.  My guess is that
on some boots, we are writing this value from cpu1
and that confuses SMM.  It is likely that booting
with HT disabled in the BIOS or with "maxcpus=1"
or a UP kernel will make the failure go away.
(give it a try)

But why did 2.6.20 work and 2.6.21-rc7 fails?
It turns out that for FADT revisions before r3,
2.6.20 would explicitly clear this field when
converting the BIOS supplied FADT into the
Linux' in-memory ACPI 2.0 format.

Upon the table-re-write in 2.6.21, we simply bcopy
the entire BIOS supplied FADT into a buffer, preserving
this field, and exposing this series of BIOS bugs.
Comment 13 Len Brown 2007-04-25 21:29:36 UTC
Created attachment 11275 [details]
patch vs 2.6.21-rc7

Please test this patch.
On this system, it should fix the boot issue
and result in an addition line in dmesg.
Comment 14 Olaf Kirch 2007-04-27 23:55:33 UTC
Thanks, Len - this patch fixes the crashes I've been seeing,
and the printk triggers.
Comment 15 Len Brown 2007-04-28 18:01:29 UTC
Created attachment 11318 [details]
patch for 2.6.22-git1

Thanks for testing Olaf.
patch in comment #13 sent to 2.6.21.stable
Attached is a slightly expanded patch w/o prink for 2.6.22.
Comment 16 Len Brown 2007-04-30 20:38:29 UTC
patch in comment #15 shipped post 2.6.22-git3, expected -git4
closed.

Note You need to log in before you can comment on or make changes to this bug.