Bug 33012 - computer restart without warning
computer restart without warning
Status: CLOSED CODE_FIX
Product: Platform Specific/Hardware
Classification: Unclassified
Component: x86-64
All Linux
: P1 blocking
Assigned To: platform_x86_64@kernel-bugs.osdl.org
:
Depends on:
Blocks: 32012
  Show dependency treegraph
 
Reported: 2011-04-11 02:47 UTC by Alexandre Demers
Modified: 2011-05-30 07:48 UTC (History)
9 users (show)

See Also:
Kernel Version: 2.6.39-rc2
Tree: Mainline
Regression: Yes


Attachments
.config at bad commit (113.97 KB, application/octet-stream)
2011-04-13 03:02 UTC, Alexandre Demers
Details
dmesg from 2.6.38.2 (122.71 KB, application/octet-stream)
2011-04-13 04:09 UTC, Alexandre Demers
Details
new dmesg (122.77 KB, application/octet-stream)
2011-04-13 14:57 UTC, Alexandre Demers
Details
Longer dmesg (444.61 KB, application/octet-stream)
2011-04-13 17:25 UTC, Alexandre Demers
Details
dmesg from bad commit WITH proposed fix (64.44 KB, application/octet-stream)
2011-04-13 23:24 UTC, Alexandre Demers
Details

Description Alexandre Demers 2011-04-11 02:47:20 UTC
Using a tri-core AMD 64bit CPU.

With kernel 2.6.39-rc2, my system reboots everytime. Bisecting seems to point commit f005fe12b90c5b9fe180a09209a893e09affa8aa x86-64: Move out cleanup higmap [_brk_end, _end) out of init_memory_mapping().

I still have to test the latest git from today and also to sync to rc2 and revert that commit to confirm I'm pointing at the culprit commit.

I was not having that problem with 2.6.38.

I will be back soon (Monday or Tuesday) with a confirmation.
Comment 1 Alexandre Demers 2011-04-11 03:04:34 UTC
Oops, pointing to wrong commit (by one commit in the tree). The first bad commit identified is d2137d5af4259f50c19addb8246a186c9ffac325 Merge branch 'linus' into x86/bootmem

Author Ingo Molnar<mingo@elte.hu>
Author date 11-02-14 5:55 AM
Parent x86-64: Move out cleanup higmap [_brk_end, _end) out of i...
Parent klist: Fix object alignment on 64-bit.
Child Merge branch 'x86/bootmem' into x86/mm
Branch origin/master (Merge branch 'fbdev-fixes-for-linus' of git://git.kernel....) 
Branch master (Linux 2.6.39-rc1) 
Follows v2.6.38-rc4 (Linux 2.6.38-rc4)
Precedes v2.6.39-rc1 (Linux 2.6.39-rc1)
Comment 2 Alexandre Demers 2011-04-11 22:15:39 UTC
I confirm the culprit commit d2137d5af4259f50c19addb8246a186c9ffac325. I will be testing the latest git version to see if it has been fixed.
Comment 3 Rafael J. Wysocki 2011-04-11 23:07:55 UTC
First-Bad-Commit : d2137d5af4259f50c19addb8246a186c9ffac325
Comment 4 David Rientjes 2011-04-11 23:15:39 UTC
Suggest earlyprintk= for further diagnosis.
Comment 5 Alexandre Demers 2011-04-11 23:41:17 UTC
did you mean earlyprintk=1?
Comment 6 David Rientjes 2011-04-11 23:46:05 UTC
No, you need to specify its route, which is probably over serial or tty.  Try

	earlyprintk=ttyS0,115200n8

first.

See the earlyprintk= entry in Documentation/kernel-parameters.txt for more information.
Comment 7 Alexandre Demers 2011-04-12 13:59:40 UTC
Still present in rc3. I'll try the earlyprintk parameter.
Comment 8 Yinghai Lu 2011-04-12 20:28:42 UTC
can you please send out .config and working dmesg ?
Comment 9 Alexandre Demers 2011-04-13 03:02:40 UTC
Created attachment 54202 [details]
.config at bad commit

This is the .config file used when compiling kernel with bad commit. Dmesg from 2.6.38.2 will follow.
Comment 10 Alexandre Demers 2011-04-13 03:16:12 UTC
(In reply to comment #9)
> Created an attachment (id=54202) [details]
> .config at bad commit
> 
> This is the .config file used when compiling kernel with bad commit. Dmesg from
> 2.6.38.2 will follow.

I should have said "when hard resetting at bad commit"
Comment 11 Alexandre Demers 2011-04-13 04:09:20 UTC
Created attachment 54222 [details]
dmesg from 2.6.38.2

Here is dmesg from the latest good version I've been using (2.6.38.2)
Comment 12 Alexandre Demers 2011-04-13 04:15:28 UTC
I can't see anything special using earlyprintk (ttyS0,115200n8 nor vga). It resets near the "sd 9:0:0:0: [sdc] Assuming drive cache: write through". Maybe I'm not using it correctly. Should I specify something else. Documentation about the parameter is pretty short.
Comment 13 Yinghai Lu 2011-04-13 05:50:26 UTC
that dmesg seem get clipped ...

can you check /var/log/messages or /var/log/bootlog or /var/log/dmesg?

that should have complete boot log.
Comment 14 Yinghai Lu 2011-04-13 07:28:15 UTC
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y

are you using xen/dom0, or use it on real HW directly?
Comment 15 Alexandre Demers 2011-04-13 14:40:33 UTC
(In reply to comment #14)
> CONFIG_XEN=y
> CONFIG_XEN_DOM0=y
> CONFIG_XEN_PRIVILEGED_GUEST=y
> CONFIG_XEN_PVHVM=y
> 
> are you using xen/dom0, or use it on real HW directly?

I'm not using Xen, it's there just in case I'd like to use it at some point.
Comment 16 Alexandre Demers 2011-04-13 14:57:17 UTC
Created attachment 54262 [details]
new dmesg

taken directly from /var/log/dmesg. It seems similar to the previous one.
Comment 17 Yinghai Lu 2011-04-13 16:28:32 UTC
You may need to increase

CONFIG_LOG_BUF_SHIFT=17

to

CONFIG_LOG_BUF_SHIFT=21

to increase the buffer

or boot with

log_buf_len=2M or even more.
Comment 18 Alexandre Demers 2011-04-13 17:25:25 UTC
Created attachment 54302 [details]
Longer dmesg

increased log buffer lenght. Now you have a complete dmesg log from 2.6.38.2 (latest good kernel)
Comment 19 Alexandre Demers 2011-04-13 18:59:33 UTC
http://lists.freedesktop.org/archives/dri-devel/2011-April/010133.html to be followed on the dri-devel list.
Comment 20 Yinghai Lu 2011-04-13 19:23:07 UTC
On 04/13/2011 10:21 AM, Joerg Roedel wrote:
> > On Wed, Apr 13, 2011 at 08:46:09AM +0200, Ingo Molnar wrote:
> > First of all, I bisected between v2.6.37-rc2..f005fe12b90c which where
> > only a couple of patches and merged v2.6.38-rc4 in at every step. There
> > was no failure found.
> > Then I tried this again, but this time I merged v2.6.38-rc5 at every
> > step and was successful. The bad commit in this branch turned out to be
> > 
> > 	1a4a678b12c84db9ae5dce424e0e97f0559bb57c
> > 
> > which is related to memblock.
> > 
> > Then I tried to find out which change between 2.6.38-rc4 and 2.6.38-rc5
> > is needed to trigger the failure, so I used f005fe12b90c as a base,
> > bisected between v2.6.38-rc4..v2.6.38-rc5 and merged every bisect step
> > into the base and tested. Here the bad commit turned out to be
> > 
> > 	e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20
> > 
> > which is related to gart. It turned out that the gart aperture on that
> > box is on another position with these patches. Before it was as
> > 0xa4000000 and now it is at 0xa0000000. It seems like this has something
> > to do with the root-cause.
> > 
> > Reverting commit 1a4a678b12c84db9ae5dce424e0e97f0559bb57c fixes the
> > problem btw. and booting with iommu=soft also works, but I have no idea
> > yet why the aperture at that address is a problem (with the patch
> > reverted the aperture lands at 0x80000000).
> > 
> > I have put some debug-data online. There is my .config and two
> > dmesg-files for good (==2.6.39-rc3 + revert) and bad (==2.6.39-rc3)
> > I also created these dmesg-files again with memblock=debug, maybe that
> > helps to find the problem. The files are at
> > 
> > 	http://www.8bytes.org/~joro/debug/
thanks for the bisecting...

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

but your radeon does use [0xa0000000, 0xbfffffff)

[    4.281993] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.290672] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.298550] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.309857] [drm] RAM width 32bits DDR
[    4.313748] [TTM] Zone  kernel: Available graphics memory: 1896524 kiB.
[    4.320379] [TTM] Initializing pool allocator.
[    4.324948] [drm] radeon: 320M of VRAM memory ready
[    4.329832] [drm] radeon: 512M of GTT memory ready.

and the one seems working:

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0x80000000-0x83ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ 80000000
[    0.000000]     memblock_x86_reserve_range: [0xacb6bdc0-0xacb6bddf]          BOOTMEM

will use different position...

[    4.250159] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.258830] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.266742] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.271549] [drm] RAM width 32bits DDR
[    4.275435] [TTM] Zone  kernel: Available graphics memory: 1896526 kiB.
[    4.282066] [TTM] Initializing pool allocator.
[    4.282085] usb 7-2: new full speed USB device number 2 using ohci_hcd
[    4.293076] [drm] radeon: 320M of VRAM memory ready
[    4.298277] [drm] radeon: 512M of GTT memory ready.
[    4.303218] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    4.309854] [drm] Driver supports precise vblank timestamp query.
[    4.315970] [drm] radeon: irq initialized.
[    4.320094] [drm] GART: num cpu pages 131072, num gpu pages 131072

So question is why radeon is using the address [0xa0000000 - 0xc000000], and in E820 it is RAM ....

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000acb8d000 (usable)
[    0.000000]  BIOS-e820: 00000000acb8d000 - 00000000acb8f000 (reserved)
[    0.000000]  BIOS-e820: 00000000acb8f000 - 00000000afce9000 (usable)
[    0.000000]  BIOS-e820: 00000000afce9000 - 00000000afd21000 (reserved)
[    0.000000]  BIOS-e820: 00000000afd21000 - 00000000afd4f000 (usable)
[    0.000000]  BIOS-e820: 00000000afd4f000 - 00000000afdcf000 (reserved)
[    0.000000]  BIOS-e820: 00000000afdcf000 - 00000000afecf000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000afecf000 - 00000000afeff000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000afeff000 - 00000000aff00000 (usable)


so looks bios program wrong address to the radon card?
Comment 21 Joerg Roedel 2011-04-13 19:48:26 UTC
On Wed, Apr 13, 2011 at 12:14:55PM -0700, Yinghai Lu wrote:
> thanks for the bisecting...
>
> so those two patches uncover some problems.
>
> [    0.000000] Checking aperture...
> [    0.000000] No AGP bridge found
> [    0.000000] Node 0: aperture @ a0000000 size 32 MB
> [    0.000000] Aperture pointing to e820 RAM. Ignoring.
> [    0.000000] Your BIOS doesn't leave a aperture memory hole
> [    0.000000] Please enable the IOMMU option in the BIOS setup
> [    0.000000] This costs you 64 MB of RAM
> [    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
+> [    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000
>
> so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

It is actually beyond 4GB on that machine, this value read here is from
the previous kernel-boot. The BIOS does not reset these values on a
reboot.

> but your radeon does use [0xa0000000, 0xbfffffff)

Yes, I suspected that too (and spent a few hours reading radeon code),
but then I talked the Alex Deucher and he explained that these addresses
which the driver prints for GTT and VRAM are in the GPU address space
and do not refer to system ram. So this shouldn't be the problem.
Comment 22 Yinghai Lu 2011-04-13 21:30:06 UTC
Alexandre's working version on 2.6.38.2 is using

Checking aperture...
No AGP bridge found
Node 0: aperture @ a4000000 size 32 MB
Aperture pointing to e820 RAM. Ignoring.
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ a4000000
PM: Registered nosave memory: 00000000a4000000 - 00000000a8000000

so it is even with 512M alignment.
Comment 23 Alexandre Demers 2011-04-13 23:24:42 UTC
Created attachment 54322 [details]
dmesg from bad commit WITH proposed fix

I applied the proposed fix and I was able to boot properly. This is the output from dmesg.
Comment 24 Alexandre Demers 2011-04-13 23:26:03 UTC
I applied the proposed fix (in the email discussion) and it fixed the boot process.
Comment 25 H. Peter Anvin 2011-04-13 23:27:50 UTC
Could you please specify which exact proposed fix that was?
Comment 26 Yinghai Lu 2011-04-13 23:33:41 UTC
Checking aperture...
No AGP bridge found
Node 0: aperture @ 80000000 size 32 MB
Aperture pointing to e820 RAM. Ignoring.
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 80000000
PM: Registered nosave memory: 0000000080000000 - 0000000084000000

looks like the big alignment to 1g.
Comment 27 Yinghai Lu 2011-04-13 23:37:44 UTC
looks like any address other than 0xa0000000 will work.
Comment 28 H. Peter Anvin 2011-04-13 23:38:59 UTC
Okay, so the question is why, and how to prevent it by design, rather than accident.
Comment 29 Alexandre Demers 2011-04-14 00:13:09 UTC
Sorry, just for the info since there is a parallel discussion on the dri mailing list, here is the discussion and proposed fix:
http://lists.freedesktop.org/archives/dri-devel/2011-April/010150.html


diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
index 86d1ad4..3b6a9d5 100644
--- a/arch/x86/kernel/aperture_64.c
+++ b/arch/x86/kernel/aperture_64.c
@@ -83,7 +83,7 @@ static u32 __init allocate_aperture(void)
      * so don't use 512M below as gart iommu, leave the space for kernel
      * code for safe
      */
-    addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+    addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);
     if (addr == MEMBLOCK_ERROR || addr + aper_size > 0xffffffff) {
         printk(KERN_ERR
             "Cannot allocate aperture memory hole (%lx,%uK)\n",


I'm now about to test the fix against 2.6.39-rc3.
Comment 30 Alexandre Demers 2011-04-14 01:46:06 UTC
Tested on rc3 and it works fine here with the "fix".
Comment 31 Yinghai Lu 2011-04-14 06:17:16 UTC
can you please try only patch2 and patch1+patch2?

want to know what happen if k8 gart and radeon gtt are all set to 0x80000000

diff --git a/drivers/gpu/drm/radeon/radeon_device.c b/drivers/gpu/drm/radeon/radeon_device.c
index 890217e..08e1961 100644
--- a/drivers/gpu/drm/radeon/radeon_device.c
+++ b/drivers/gpu/drm/radeon/radeon_device.c
@@ -325,6 +325,8 @@ void radeon_gtt_location(struct radeon_device *rdev, struct radeon_mc *mc)
 			mc->gtt_size = size_bf;
 		}
 		mc->gtt_start = (mc->vram_start & ~mc->gtt_base_align) - mc->gtt_size;
+		if (mc->gtt_start == 0xa0000000)
+			mc->gtt_start = 0x80000000;
 	} else {
 		if (mc->gtt_size > size_af) {
 			dev_warn(rdev->dev, "limiting GTT\n");
Comment 32 Alexandre Demers 2011-04-14 15:01:01 UTC
Tested:

Applying both patches make the system resets. Applying only one of the patches allows it to boot properly.
Comment 33 Joerg Roedel 2011-04-15 07:00:07 UTC
(In reply to comment #32)
> Applying both patches make the system resets. Applying only one of the patches
> allows it to boot properly.

Makes sense. When you apply both patches the both apertures end up to be on the same address (0x80000000) again.

Btw. can you please post the exact model of your internal graphics card in that box? Is it a dedicated card (with on-card memory) or part of the chipset (and uses cpu ram with no own memory).
Comment 34 herrmann.der.user 2011-04-15 14:23:05 UTC
Alexandre,

can you please run on your system the command (as root)

# lsmsr MC4_CTL

The command is part of the x86info package.
It would be good to know whether GART TLB Walk Errors are masked or not on your system.

If it's already masked Joerg's patch
(http://marc.info/?l=linux-kernel&m=130287312009403&w=2)
is of no use on your system.

Thanks.
Comment 35 Alexandre Demers 2011-04-15 14:28:35 UTC
(In reply to comment #33)
> (In reply to comment #32)
> > Applying both patches make the system resets. Applying only one of the patches
> > allows it to boot properly.
> 
> Makes sense. When you apply both patches the both apertures end up to be on the
> same address (0x80000000) again.
> 
> Btw. can you please post the exact model of your internal graphics card in that
> box? Is it a dedicated card (with on-card memory) or part of the chipset (and
> uses cpu ram with no own memory).

I'm using the following model:
RS780 (HD3200) integrated

If you want more info on the mainboard:
http://www.gigabyte.com/products/product-page.aspx?pid=2814#ov
Comment 36 Alexandre Demers 2011-04-15 17:24:34 UTC
(In reply to comment #34)
> Alexandre,
> 
> can you please run on your system the command (as root)
> 
> # lsmsr MC4_CTL
> 
> The command is part of the x86info package.
> It would be good to know whether GART TLB Walk Errors are masked or not on your
> system.
> 
> If it's already masked Joerg's patch
> (http://marc.info/?l=linux-kernel&m=130287312009403&w=2)
> is of no use on your system.
> 
> Thanks.

Sorry, it took some time, I was in the middle of some updates and something went wrong. Then, I installed the x86info provided by Ubuntu, but lsmsr was missing (I have to open a bug about it). So I downloaded the source from the project's website, but latest version can't be compiled becausse of an error (I have to open a bug about it also). I'm using version 1.28, since it's the last working version I was able to compile. So now about the command:
./lsmsr MC4_CTL
MC4_CTL              = 0x000000003fffffff
MC4_CTL_MASK         = 0x0000000000780000


./lsmsr MC4_CTL -V3 | grep "MC4\|Gart"
MC4_CTL              = 0x000000003fffffff
  GartTblWkEn=0x1
MC4_CTL_MASK         = 0x0000000000780000
  GartTblWkEn=0

So it looks really similar to the results of Joerg.

I'll be now testing the suggested patch.
Comment 37 Alexandre Demers 2011-04-15 18:58:08 UTC
Applied Joerg's patch and it now works correctly. It seems to be the good fix for this processor/BIOS problem.
Comment 39 Török Edwin 2011-05-15 14:06:49 UTC
Apparently the fix for this bug introduced a regression when run under KVM:
https://bugzilla.kernel.org/show_bug.cgi?id=35132
Comment 40 Florian Mickler 2011-05-30 07:48:29 UTC
A patch referencing this bug report has been merged in v3.0-rc1:

commit d47cc0db8fd6011de2248df505fc34990b7451bf
Author: Roedel, Joerg <Joerg.Roedel@amd.com>
Date:   Thu May 19 11:13:39 2011 +0200

    x86, amd: Use _safe() msr access for GartTlbWlk disable code

Note You need to log in before you can comment on or make changes to this bug.