Bug 11676

Created attachment 18114 [details]
Successful boot, 2.6.27-rc8, w/ iommu=soft

Created attachment 18115 [details]
Successful boot, 2.6.26.5, using the gart iommu

Created attachment 18116 [details]
2.6.27-rc8 .config

Created attachment 18125 [details]
Failed boot, what's left on screen, ~ 40 lines

OK, I went ahead and did a reboot using the gart iommu, and copied down what was left on screen at the freeze.  Here it is.

Note the comment lines at the beginning and end.  This is NOT intended to be char-4-char literal, tho I tried to get the addresses literal in case they are useful, but only to allow you to sequence and compare the lines with the successful boots.  Of course, I was using vga=ask (and setting bios mode 133) instead of the normal radeon framebuffer, in ordered to get any on-screen output at all.

Can you please try to bisect the problem between 2.6.26 and 2.6.27-rc2?
For information about how to bisect please look at

http://www.kernel.org/doc/local/git-quick.html#bisect

Please give us the commit if you have success.

Yinghai,

you changed some code in the GART aperture and initialization code. Can you also have a look at this problem?

please try to boot with "debug"...

(In reply to comment #7)
> please try to boot with "debug"...

As a command line option or a kconfig option?

Meanwhile, I've bisected some way.  It's early in the cycle, between 2.6.26 and 2.6.26-git4.  I'm back online to get the in-between snapshots and check bug status.

FWIW, those early gits (I tried 4, 9, and 14) seem to be even worse, if anything.  Even vga=ask, then mode 6 (80x60 vga, IIRC) seems to fail, and while I'd occasionally get a freeze by the rc8 I was just testing, thus allowing me to actually write down the screen of messages at that point, those early gits don't seem to freeze at all.  They just reboot immediately, so fast that I had trouble verifying it was rebooting at the same spot.  I finally glimpsed a 256 flashing by right before the reboot, and I get the same no output at all with the framebuffer, so it appears to be real close to the same spot in any case.

OK, it's nailed down to the delta between 2.6.26 and 2.6.26-git1.  I did have to turn off the tg3 NIC option to get git1 to install properly, but that should be something different and was fixed by git2.  Just to be sure it wasn't something changed in my setup since I last compiled 2.6.26, I recompiled and reinstalled it, and it booted fine, so yes indeed, it's a commit in that first day post 2.6.26.  (Well, unless I screwed up my choices in make oldconfig, of course, but I've tried all the differences I thought might matter, no difference.)

... And with that I'm out for the day... I'll try further bisection and the debug thing later.

just tried tip/master with s2885 + 8g + nvidia display card..
it works well

LBSuse:~ # grep -i aperture dmesg.txt
Checking aperture...
Aperture from AGP @ c0000000 old size 32 MB
Aperture from AGP @ c0000000 size 512 MB (APSIZE e00)
Node 0: aperture @ c0000000 size 512 MB
Node 1: aperture @ c0000000 size 512 MB
agpgart-amd64 0000:04:00.0: AGP aperture is 512M @ 0xc0000000
PCI-DMA: Reserving 256MB of IOMMU area in the AGP aperture
LBSuse:~ # cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009fbff : System RAM
0009fc00-0009ffff : reserved
000a0000-000bffff : PCI Bus #04
000c0000-000dffff : pnp 00:0d
000e0000-000fffff : reserved
00100000-9bfeffff : System RAM
  00200000-0096dc83 : Kernel code
  0096dc84-00ea89c7 : Kernel data
  00fcd000-011d4ae3 : Kernel bss
9bff0000-9bffefff : ACPI Tables
9bfff000-9bffffff : ACPI Non-volatile Storage
9c000000-9c4fffff : PCI Bus #00
  9c300000-9c3fffff : PCI Bus 0000:02
9c500000-fc5fffff : PCI Bus #04
  9c500000-bc4fffff : PCI Bus 0000:05
    a0000000-afffffff : 0000:05:00.0
  c0000000-dfffffff : GART
    c0000000-dfffffff : aperture
fc600000-fc9fffff : PCI Bus #00
  fc600000-fc7fffff : PCI Bus 0000:01
    fc700000-fc77ffff : 0000:01:0b.0
    fc7f8000-fc7fbfff : 0000:01:0c.0
    fc7fd000-fc7fdfff : 0000:01:00.0
      fc7fd000-fc7fdfff : ohci_hcd
    fc7fe000-fc7fefff : 0000:01:00.1
      fc7fe000-fc7fefff : ohci_hcd
    fc7ff000-fc7ff7ff : 0000:01:0c.0
      fc7ff000-fc7ff7ff : ohci1394
    fc7ffc00-fc7fffff : 0000:01:0b.0
      fc7ffc00-fc7fffff : sata_sil
  fc800000-fc8fffff : PCI Bus 0000:02
    fc8e0000-fc8effff : 0000:02:09.0
    fc8f0000-fc8fffff : 0000:02:09.0
      fc8f0000-fc8fffff : tg3
  fc9fe000-fc9fefff : IOAPIC 2
    fc9fe000-fc9fefff : 0000:00:0b.1
  fc9ff000-fc9fffff : IOAPIC 1
    fc9ff000-fc9fffff : 0000:00:0a.1
fca00000-febfffff : PCI Bus #04
  fca00000-feafffff : PCI Bus 0000:05
    fd000000-fdffffff : 0000:05:00.0
    feae0000-feafffff : 0000:05:00.0
fec00000-ffffffff : PCI Bus #00
  fec00000-fec00fff : IOAPIC 0
    fec00000-fec00fff : pnp 00:0c
  fec01000-fec013ff : HPET 0
  fee00000-fee00fff : Local APIC
    fee00000-fee00fff : pnp 00:0c
  ff780000-ffffffff : reserved
100000000-1ffffffff : System RAM
200000000-fcffffffff : PCI Bus #00

-+-[0000:04]-+-00.0  Advanced Micro Devices [AMD] AMD-8151 System Controller
 |           \-01.0-[0000:05]----00.0  nVidia Corporation NV34 [GeForce FX 5500]
 \-[0000:00]-+-06.0-[0000:01]--+-00.0  Advanced Micro Devices [AMD] AMD-8111 USB
             |                 +-00.1  Advanced Micro Devices [AMD] AMD-8111 USB
             |                 +-0b.0  Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller
             |                 \-0c.0  Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
             +-07.0  Advanced Micro Devices [AMD] AMD-8111 LPC
             +-07.1  Advanced Micro Devices [AMD] AMD-8111 IDE
             +-07.2  Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0
             +-07.3  Advanced Micro Devices [AMD] AMD-8111 ACPI
             +-07.5  Advanced Micro Devices [AMD] AMD-8111 AC97 Audio
             +-0a.0-[0000:02]----09.0  Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet
             +-0a.1  Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC
             +-0b.0-[0000:03]--
             +-0b.1  Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC
             +-18.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
             +-18.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
             +-18.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
             +-18.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
             +-19.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
             +-19.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
             +-19.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
             \-19.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control

only difference that you are using ati card...

please try tip/master

http://people.redhat.com/mingo/tip.git/readme.txt

> OK, it's nailed down to the delta between 2.6.26 and 2.6.26-git1. 
[...]

2.6.26-git1 is 50515af207d410c9f228380e529c56f43c3de0bd.

The v2.6.26..50515af207d410c9f2 range is ~2000 commits, and all the x86 
updates are in there.

	Ingo

The GART aperture is not covered by E820 (not marked as reserved or usable). May this be a problem?

(In reply to comment #10)
> just tried tip/master with s2885 + 8g + nvidia display card..
> it works well

> LBSuse:~ # cat /proc/iomem

> only difference that you are using ati card...

Are you using a framebuffer or the standard vgacon?  I'm using Radeon framebuffer here, and with vga=ask, then mode 1 (or similar vga mode), on a kernel that works, it seems to switch to framebuffer right about the point at which it's crashing on one that doesn't.  To be sure, this isn't really going by log, but by "feel".  Should I try with it configured out?

And would my /proc/iomem be of help also?  Should I post the .27-rc iommu=soft or the old .26 gart version, or both?

FWIW, my schedule over the next 24 hours isn't good for working on this that much, but the 24-36 after that, Friday and half of Saturday my time, is better...  I know time is compressed due to the release cycle point we're at.

Response to #13: that is normal; the GART aperture is usually identified via a BAR rather than a static resource.

yes. only require gart aperture is not overlapped with e820 ram areas.

enabld fb, it still works well.

ARRGG!

I had started working on the bisect, but let's just say I brown-paper-bagged something... and I'm glad I have tested-working backups!  The bisect data appears to be safe, tho.  Here's the log I have so far.  I think it was < 100 commits left to test, so that should narrow it down significantly:

git bisect start
# bad: [50515af207d410c9f228380e529c56f43c3de0bd] firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR
git bisect bad 50515af207d410c9f228380e529c56f43c3de0bd
# good: [bce7f793daec3e65ec5c5705d2457b81fe7b5725] Linux 2.6.26
git bisect good bce7f793daec3e65ec5c5705d2457b81fe7b5725
# bad: [d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8] Merge branch 'sched/for-linus' into tracing/for-linus
git bisect bad d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8
# good: [3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce] Merge branch 'x86/mpparse' into x86/devel
git bisect good 3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce
# good: [e93ef949fd9a3f237aedfb8e64414b28980530b8] x86: rename paravirtualized TSC functions
git bisect good e93ef949fd9a3f237aedfb8e64414b28980530b8
# good: [b6770c83b4b88dd832cc77916b1935b0311c64e0] x86, VisWS: do not allow VisWS for Voyager
git bisect good b6770c83b4b88dd832cc77916b1935b0311c64e0

(The paravert TSC, e93ef949f..., was a shot in the dark, but I got another in before the incident to confirm it, so...)

I really gotta sleep now before I brown-paper-bag anything else.  After that it'll probably take a few hours figuring out and restoring status, before I can get back to this.  Hopefully tomorrow.

* bugme-daemon@bugzilla.kernel.org <bugme-daemon@bugzilla.kernel.org> wrote:

> I really gotta sleep now before I brown-paper-bag anything else.  
> After that it'll probably take a few hours figuring out and restoring 
> status, before I can get back to this.  Hopefully tomorrow.

yes, the bisection seems to start to point in the right direction. 
Suspect commits are:

2387ce5: x86: make 64bit hpet_set_mapping to use ioremap too, v2
87a1c44: x86: get x86_phys_bits early
32b23e9: x86: max_low_pfn_mapped fix #4
7b479be: x86, e820: remove end_user_pfn
9958e81: x86: max_low_pfn_mapped fix, #3
965194c: x86: max_low_pfn_mapped fix, #2
7ab073b: x86: max_low_pfn_mapped fix, #1
69a7704: x86: e820: user-defined memory maps: remove the range instead of update
a361ee5: x86: fix /dev/mem compatibility under PAT

but any of the other x86 commits could be the culprit too.

here's your current bisection log, in a copy&paste-able form:

git bisect start
git bisect bad 50515af207d410c9f228380e529c56f43c3de0bd
git bisect good bce7f793daec3e65ec5c5705d2457b81fe7b5725
git bisect bad d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8
git bisect good 3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce
git bisect good e93ef949fd9a3f237aedfb8e64414b28980530b8
git bisect good b6770c83b4b88dd832cc77916b1935b0311c64e0

	Ingo

Created attachment 18164 [details]
full bisect log

OK, completed the bisect, but is there a way to spit out the result again, without a replay?  Here's the log in any case, and a replay spits out the following:

f361a450bf1ad14e2b003217dbf3958638631265 is first bad commit
commit f361a450bf1ad14e2b003217dbf3958638631265
Author: Yinghai Lu <yhlu.kernel@gmail.com>
Date:   Thu Jul 10 20:38:26 2008 -0700

    x86: introduce max_low_pfn_mapped for 64-bit

    when more than 4g memory is installed, don't map the big hole below 4g.

    Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
    Cc: Suresh Siddha <suresh.b.siddha@intel.com>
    Signed-off-by: Ingo Molnar <mingo@elte.hu>

:040000 040000 c8d1071138c28eaf8544f3d956a573cf0ad4bc84 c866895ac7917c7327058e3ae5de677d0eaaacf1 M      arch
:040000 040000 9bb54b21a753af68a5e6e7ffff4cf18678d179c0 7e39996b3df55d851fabef79de351c0a431cf3cc M      include

Reply-To: yinghai@kernel.org

On Sat, Oct 4, 2008 at 1:40 PM,  <bugme-daemon@bugzilla.kernel.org> wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=11676
>
>
>
>
>
> ------- Comment #20 from 1i5t5.duncan@cox.net  2008-10-04 13:40 -------
> Created an attachment (id=18164)
>  --> (http://bugzilla.kernel.org/attachment.cgi?id=18164&action=view)
> full bisect log
>
> OK, completed the bisect, but is there a way to spit out the result again,
> without a replay?  Here's the log in any case, and a replay spits out the
> following:
>
> f361a450bf1ad14e2b003217dbf3958638631265 is first bad commit
> commit f361a450bf1ad14e2b003217dbf3958638631265
> Author: Yinghai Lu <yhlu.kernel@gmail.com>
> Date:   Thu Jul 10 20:38:26 2008 -0700
>
>    x86: introduce max_low_pfn_mapped for 64-bit
>
>    when more than 4g memory is installed, don't map the big hole below 4g.
>
>    Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
>    Cc: Suresh Siddha <suresh.b.siddha@intel.com>
>    Signed-off-by: Ingo Molnar <mingo@elte.hu>
>
> :040000 040000 c8d1071138c28eaf8544f3d956a573cf0ad4bc84
> c866895ac7917c7327058e3ae5de677d0eaaacf1 M      arch
> :040000 040000 9bb54b21a753af68a5e6e7ffff4cf18678d179c0
> 7e39996b3df55d851fabef79de351c0a431cf3cc M      include
>
>
could be ati fb want to use direct map address?

YH

Created attachment 18165 [details]
direct map that range when agp is there

please try the patch in comment #22

> could be ati fb want to use direct map address?

would be nice to figure out where that happens. Access not via ioremap() 
is not allowed in general - incidental mapping of some physical memory 
range in previous kernels might have masked a bug.

> please try the patch in comment #22

or try tip/master:

  http://people.redhat.com/mingo/tip.git/README

which has this fix included.

	Ingo

The patch is a confirmed fix on linus/master (v2.6.27-rc8-84-gfec6ed1).

As for tip, I'll try it, but FWIW here's a bit of feedback.  For likely git novices, it may be worth considering whether alternate tree suggestions are worth the potential confusion, or at least get the linked documentation coordinated so it doesn't conflict, thus causing confusion.

Up-bug/thread, I was asked to do bisect, then tip/master was suggested.  Both suggestions pointed at helpful on-their-own instructions, but the bisect aka git-quick document suggested git-clone, while the tip.git/README doc suggests git remote and etc (yes, I know the tip README isn't for quite the same audience, perhaps a different doc to point testers to might be useful).  This was quite confusing and it took me several hours of additional testing time to read up and absorb enough git to resolve the conflicts.  I eventually used git remote add to add both, then decided that bisect was the most likely fruitful way to accurately pin-down the problem, since tip has other patches that would have potentially complicated the picture.

Resolving this confusion, either by coordinating the docs, or by plainly prioritizing competing suggestions (ie: "Bisect will likely be the most direct route to a fix, but if you aren't comfortable trying that, here's an alternate suggestion."), but preferably both, will make things significantly easier for others taking a path similar to the one I just traveled.  The result will be getting test results sooner, or possibly getting them at all, since the cost of confusion may be someone dropping the report entirely, possibly scaring off what might otherwise turn into a regular tester, when they decide it's just too confusing and must be beyond them.

Hope it helps...

> [...]  This was quite confusing and it took me several hours of 
> additional testing time to read up and absorb enough git to resolve 
> the conflicts. [...]

hm, the -tip readme says:

|  #
|  # if you need to do bisection of the -tip tree, then do:
|  # (but first check that linus/master is indeed a 'good' kernel :-)
|  #
|  git bisect start
|  git bisect good linus/master
|  git bisect bad tip/master

and that is still 100% accurate to bisect bugs that only show up in 
tip/master.

If you want to debug bugs in mainline, you can still use the symbols 
referenced to in the -tip readme:

  git bisect start
  git bisect bad linus/master
  git bisect good v2.6.24     # last known good kernel

So ... it would be very useful if you could suggest a few lines of extra 
info to tip's readme that would have reduced your confusion. It's meant 
for git newbies too, so if it causes confusion that's unintentional. 

Maybe send a patch to that file if you dont mind? Such readmes are hard 
to write because _we_ are not newbies anymore so we dont notice the 
hickups in its logic. We very much rely on feedback like yours to 
improve it.

Thanks!

Confirming tip/master (tip-history-2008-09-09_09.14_Tue-1196-g6ed3e1f) works as well, as expected since it has the patch.

Interesting additional config options.  I liked the raid-autodetect option, since I turn that off on the kernel command line since I use mdp, and the autodetect loads unpartitioned md instead.  Additionally, the built-in kernel command line could be helpful given the length of mine, tho I elected not to use it ATM, since I still consider linus/mainline my usual kernel.  It'd be very nice to have both of these in mainline at some point.

... Anyway, now that both tip and mainline-plus-patch are working, I can try some of the other new .27 features, like mtrr cleanup, that I kept off while trying to trace this bug down. =:^)

(In reply to comment #27)
> > [...]  This was quite confusing [...]
> 
> hm, the -tip readme says:
> 
> |  #
> |  # if you need to do bisection of the -tip tree [...]

First, I'm not sure where all this bug is gated to, but maybe this bit should head elsewhere, another bug or private mail... or doesn't that bother people here (more efficient filtering?) like it can elsewhere?

It wasn't the bisect part that was confusing, but setting up the remotes and etc in the first place.  git clone is suggested before the bisect stuff in the git-quick doc, as I mentioned, git remote add/update and related commands are suggested in the tip README.  That's what confused me.  How do they relate to each other and since the tip README was dealing with two trees while git-quick was dealing with one, how does that change things?  Further, how does one actually switch between trees?  Also, neither document is really clear on remote vs local branches.  git-quick assumes one-way-git and non-developer, tip README appears to assume more developer interest, so git-quick would be /expected/ to be simpler in that case... but what are the differences?

These are the sorts of questions I was trying to find the answers to.  The bisect part was basically fine, GETTING there was the problem. =:^)

> Maybe send a patch to that file if you dont mind? Such readmes are hard 
> to write because _we_ are not newbies anymore so we dont notice the 
> hickups in its logic. We very much rely on feedback like yours to 
> improve it.

I'm not thinking quite clearly enough on it yet to send a patch... yet.  But yes, the above you're not newbies, it's hard to write for them, was where I was headed with this too.

fixed by commit d99e90164e6cf2eb85fa94d547d6336f8127a107