Bug 11821

Summary:	System crashes/unstable with intel_iommu
Product:	Platform Specific/Hardware	Reporter:	frederik
Component:	x86-64	Assignee:	platform_x86_64 (platform_x86_64)
Status:	RESOLVED CODE_FIX
Severity:	normal	CC:	aris, dwmw2, markmc, Matt_Domsch, oleg_orel, wwlinuxengineering
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	2.6.27.3	Subsystem:
Regression:	No	Bisected commit-id:
Bug Depends on:
Bug Blocks:	56331
Attachments:	dmidecode dmesg 2.6.27.3 with intel_iommu=off dmesg Mandriva's TMB kernel based on 2.6.27-rc8 with intel_iommu acpidump Dell Latitude E6400 (BIOS version A06) dmesg with 2.6.30(.1) debug patch crash fix workaround for dcdbas driver dmesg with iommu-2.6 git kernel tree

Description frederik 2008-10-24 16:15:00 UTC

When Intel VT is enabled in the BIOS of my Dell Latitude E6400, the system becomes severely unstable and crashes during boot. I can work-around the instability with intel_iommu=off.

Depending on the precise kernel version (and of the phase of the moon?), the crashes/instability may differ. With stock 2.6.27.3:
- During the boot process, different stack traces start popping up one after another. Around the time when X is started, the machine stops responding.
- If I boot the machine in runlevel 3, stack traces continue to scroll over the screen (too fast to be able to read them). Several minutes after the series of stack traces have appeared, the machine locks up completey and scroll, caps and num lock start blinking.

I was able to read a part of the last two backtraces.
General proection fault
process: scsi_eh_0
alloc_iova_mem
alloc_iova
? fbcon_scroll
? string
iommu_alloc_iova
__intel_alloc_iova
intel_map_sg
? atomic_notificer_call_chain
ata_qc_issue
ata_exec_internal_sg
ata_exec_internal
ata_do_dev_read
? printk
ata_dev_read_id
? ahci_check_ready
? ata_wait_after_reset
? ahci_do_softreset
? ahci_check_ready
ata_dev_reread_id
....

And the last one which locked up the machine, after which the machine halted itself:
process swapper
? acpi_ec_gpe_query
acpi_os_execute
acpi_ec_gpe_handler
...

With other kernels (Mandriva's kernels), the system boots (with a backtrace when hald-addon-, but the e1000e network card timeouts after a few seconds of data transfer, needing a reboot to make it work again. With intel_iommu, the e1000e network card is perfectly stable.

Comment 1 frederik 2008-10-24 16:16:46 UTC

Created attachment 18428 [details]
dmidecode

Comment 2 frederik 2008-10-24 16:18:06 UTC

Created attachment 18429 [details]
dmesg 2.6.27.3 with intel_iommu=off

Comment 3 frederik 2008-10-24 16:23:09 UTC

Created attachment 18431 [details]
dmesg Mandriva's TMB kernel based on 2.6.27-rc8 with intel_iommu

The system boots and is somewhat usable with this kernel with intel_iommu enabled, however, there is still one backtrace when hald-addon-dell-backlight is started, an you also see the e1000e timeout error appearing in this configuration. Booting this same kernel with intel_iommu=off, makes both problems disappear.

Comment 4 Marcello Laganà 2008-10-26 04:52:02 UTC

Same problem here with the same system (DELL Latitude E6400), but appeared only when I've enabled the _Support for DMA Remapping Devices_ option in the kernel configuration (Bus Options section) in searching of ~ 200Mb of RAM lost.

With an ACPI DMAR capable system, as the Latitude is, and that option enabled, I see almost all of the RAM available but kernel goes unstable as the message "BUG: unable to handle kernel NULL pointer dereference..." claims.

I think that this problem is more related with DMAR part rather then the whole IOMMU.

Comment 5 Oleg Orel 2009-01-29 13:52:47 UTC

We are trying to make iommu working on laptops and desktop(Dell Latitude E6400, Lenovo X200 & Dell Precision T5400). However, we on all three computers have exactly the same problem. We getting interrupt with 3 type of errors reported: 

a) PTE doesn't have write bit set
b) PTE doesn't have read bit set
c) Present bit in root entry is clear


We had reported more details @ https://bugzilla.redhat.co/show_bug.cgi?id=479996

Please advice us a way to debug it, what peaces should we look at, print/check, etc...

1. Could we blame BIOS for that?
2. Is it possible that we build kernel with options what conflicting with iommu implementation in some way? If so, could you share your binary/source/config what works for you?
3. What desktop/laptops been tested and you aware that it works?
4. KVM folks, is it KVM+IOMMU works?
5. We used 2.6.28-rc2 and up-to-date git kernel. The iommu code is very different why it has changed so drastically and who are in change of that?

Oleg

Comment 6 Mark McLoughlin 2009-01-30 00:35:46 UTC

(In reply to comment #5)
> We are trying to make iommu working on laptops and desktop(Dell Latitude
> E6400,
> Lenovo X200 & Dell Precision T5400). However, we on all three computers have
> exactly the same problem. We getting interrupt with 3 type of errors
> reported: 
> 
> a) PTE doesn't have write bit set
> b) PTE doesn't have read bit set
> c) Present bit in root entry is clear

Oleg, these problems are better tracked in bug #12578

The issue in this bug is a NULL dereference in pci_find_upstream_pcie_bridge()

Comment 7 Zhang Rui 2009-06-09 03:08:16 UTC

hi, fhimpe,
can you do me a favor to get the acpidump of your dell latitude e6400 laptop?
I need this to support a new feature in Linux kernel.

BTW: you can get the latest pmtools at http://www.lesswatts.org/projects/acpi/utilities.php
build and run "acpidump > acpidump.log" to get the acpidump output.

Comment 8 frederik 2009-06-09 07:26:57 UTC

Created attachment 21824 [details]
acpidump Dell Latitude E6400 (BIOS version A06)

Comment 9 Aristeu Sergio Rozanski Filho 2009-06-14 17:02:51 UTC

not sure this is relevant, but on X200 BIOS changelog I found this:
Version 3.02
BIOS: 3.02 / ECP: 1.04

    * (Fix) Fixed an issue that might not get the Operating System started after applying the BIOS version 3.01 if both the Intel(R) Virtualization Technology setting and the Intel(R) VT-d Fearure setting are enabled under Config and CPU in the BIOS Setup Utility.

(http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=lenovo&lndocid=MIGR-70348)

Comment 10 Marcello Laganà 2009-07-04 10:55:53 UTC

Created attachment 22204 [details]
dmesg with 2.6.30(.1)

Oops remains with

commit 151ab36a2ea0b3181d103f7244636e0d16e685de
Author: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Date:   Tue Feb 17 14:14:36 2009 +0900

(Kernel >= 2.6.30)

But changed from "pci_find_upstream_pcie_bridge" to "get_domain_for_dev"

Comment 11 David Woodhouse 2009-07-04 12:12:43 UTC

Created attachment 22205 [details]
debug patch

Please could you provide output of 'lspci -vt' and dmesg with the attached patch (and with iommu enabled).

Comment 12 David Woodhouse 2009-07-04 12:35:22 UTC

Hm, this isn't even a PCI device. We shouldn't be in IOMMU code for this at all.

I suspect that we should be setting archdata.dma_ops for PCI devices only, and _not_ defaulting to the IOMMU dma_ops for all devices.

Although the dcdbas driver is arguably buggy, since it's abusing the DMA API to just allocate memory below 4GiB.

Comment 13 Matt Domsch 2009-07-04 14:05:47 UTC

Re: dcdbas.  Is there another way to allocate memory guaranteed to be below 4GB without using the DMA API?  The calls are to a "device" (though that happens to be SMM) which requires memory below 4GB.

Comment 14 David Woodhouse 2009-07-04 18:24:58 UTC

Created attachment 22207 [details]
crash fix

I think this should fix the crash, but the driver may well still not work if you have memory above 4GiB.

Comment 15 David Woodhouse 2009-07-04 18:25:44 UTC

Created attachment 22208 [details]
workaround for dcdbas driver

This might even make the offending driver work, but I don't think it's the right thing to do.

Comment 16 Marcello Laganà 2009-07-06 11:07:49 UTC

Created attachment 22231 [details]
dmesg with iommu-2.6 git kernel tree

I tried the debug patch but it doesn't add anything to the already known output. I also tried to add proposed patches to my kernel tree without success. So instead finding right kernel tree to patch or patch it by myself, I've committed directly the iommu-2.6 git branch (commit 3dfc813d94bba2046c6aed216e0fd69ac93a8e03).

Here comes dmesg log. I had to remove the inteldrm(fb) driver and using the default VESA frame buffer driver because of a useless blank screen.

Oops doesn't occur anymore but DMAR seems to fail on the Graphic Controller (also MTRR has some problems with that ... I don't know if they're related).

Comment 17 David Woodhouse 2009-07-06 11:28:31 UTC

We know the graphics driver is broken and doesn't use the DMA API properly. For now, you need to enable the "workaround broken graphics drivers" option (CONFIG_DMAR_BROKEN_GFX_WA) or boot with 'intel_iommu=igfx_off' on your command line to make it work.

The tree you tested has my patch from comment #14, but not the one from comment #15 -- since you _do_ have memory above 4GiB, the dcdbas driver might still be unreliable. I'll leave the Dell folks to work out what they want to do about that.

Thanks for testing.