|Summary:||2.6.27-rc2 to rc8, apgart fails, iommu=soft works, regression|
|Product:||Platform Specific/Hardware||Reporter:||Duncan (1i5t5.duncan)|
|Severity:||normal||CC:||bunk, herrmann.der.user, rjw, yhlu.kernel|
|Bug Depends on:|
Successful boot, 2.6.27-rc8, w/ iommu=soft
Successful boot, 126.96.36.199, using the gart iommu
Failed boot, what's left on screen, ~ 40 lines
full bisect log
direct map that range when agp is there
Description Duncan 2008-09-30 10:24:52 UTC
Latest working kernel version: 2.6.26.x Earliest failing kernel version: 2.6.27-rc2 Distribution: Gentoo/~amd64, mainline kernel Hardware Environment: Tyan s2885 mobo, dual Opteron 290s, 8 gig RAM, Software Environment: root is reiserfs on 4-spindle MDP/RAID-6, on SATA (SATA_SIL) Problem Description: Kernels from 2.6.27-rc2 thru rc8 won't boot with GART IOMMU enabled. They boot fine with iommu=soft. Thru 2.6.26.x worked fine with GART IOMMU enabled, so this is a regression. I haven't tried rc1. I normally run radeon-framebuffer @ 1920x1200, and wasn't seeing any of the kernel output, just (usually) a quite early spontaneous reboot. Using vga=ask, I found I got kernel output to the console if I used text modes, but not using graphics/vesa modes. The output stops right after it loads the GART IOMMU into the aperture, 256 meg of the 512 meg BIOS-set aperture. Very occasionally I get a set of four more lines after that, obviously 4-way matched, but no further and I seldom get that. Again, usually a spontaneous reboot, sometimes just a freeze. (I believe I have reboot on panic set, does it get read this early?) Due to an INVALID bug #11450 (my problem, not yours) earlier, I didn't get a successful compile past -rc2 until late in the game, and was busy then, thus the report this late in the cycle. I compile using gcc-4.3.1 using the following CFLAGS, which I used for 2.6.26 as well, where they work fine: CFLAGS="-march=k8 -pipe -msse3 -O2 -frename-registers -fweb -fmerge-all-constants -fgcse-sm -fgcse-las -fgcse-after-reload -ftree-vectorize -fdirectives-only -freorder-blocks-and-partition -combine" I'll attach my config and a successful boot log using iommu=soft and another from 2.6.26.x. Getting the failed boot will be harder tho I could hand-copy the lines, but as I said the last line is usually the 256 meg IOMMU assignment in the aperture, and iommu=soft works, so...
Comment 1 Duncan 2008-09-30 10:29:27 UTC
Created attachment 18114 [details] Successful boot, 2.6.27-rc8, w/ iommu=soft
Comment 2 Duncan 2008-09-30 10:34:07 UTC
Created attachment 18115 [details] Successful boot, 188.8.131.52, using the gart iommu
Comment 4 Duncan 2008-10-01 02:22:57 UTC
Created attachment 18125 [details] Failed boot, what's left on screen, ~ 40 lines OK, I went ahead and did a reboot using the gart iommu, and copied down what was left on screen at the freeze. Here it is. Note the comment lines at the beginning and end. This is NOT intended to be char-4-char literal, tho I tried to get the addresses literal in case they are useful, but only to allow you to sequence and compare the lines with the successful boots. Of course, I was using vga=ask (and setting bios mode 133) instead of the normal radeon framebuffer, in ordered to get any on-screen output at all.
Comment 5 Joerg Roedel 2008-10-01 05:10:25 UTC
Can you please try to bisect the problem between 2.6.26 and 2.6.27-rc2? For information about how to bisect please look at http://www.kernel.org/doc/local/git-quick.html#bisect Please give us the commit if you have success.
Comment 6 Joerg Roedel 2008-10-01 05:14:30 UTC
Yinghai, you changed some code in the GART aperture and initialization code. Can you also have a look at this problem?
Comment 7 Yinghai Lu 2008-10-01 08:04:30 UTC
please try to boot with "debug"...
Comment 8 Duncan 2008-10-01 09:49:48 UTC
(In reply to comment #7) > please try to boot with "debug"... As a command line option or a kconfig option? Meanwhile, I've bisected some way. It's early in the cycle, between 2.6.26 and 2.6.26-git4. I'm back online to get the in-between snapshots and check bug status. FWIW, those early gits (I tried 4, 9, and 14) seem to be even worse, if anything. Even vga=ask, then mode 6 (80x60 vga, IIRC) seems to fail, and while I'd occasionally get a freeze by the rc8 I was just testing, thus allowing me to actually write down the screen of messages at that point, those early gits don't seem to freeze at all. They just reboot immediately, so fast that I had trouble verifying it was rebooting at the same spot. I finally glimpsed a 256 flashing by right before the reboot, and I get the same no output at all with the framebuffer, so it appears to be real close to the same spot in any case.
Comment 9 Duncan 2008-10-01 11:10:49 UTC
OK, it's nailed down to the delta between 2.6.26 and 2.6.26-git1. I did have to turn off the tg3 NIC option to get git1 to install properly, but that should be something different and was fixed by git2. Just to be sure it wasn't something changed in my setup since I last compiled 2.6.26, I recompiled and reinstalled it, and it booted fine, so yes indeed, it's a commit in that first day post 2.6.26. (Well, unless I screwed up my choices in make oldconfig, of course, but I've tried all the differences I thought might matter, no difference.) ... And with that I'm out for the day... I'll try further bisection and the debug thing later.
Comment 10 Yinghai Lu 2008-10-01 23:27:44 UTC
just tried tip/master with s2885 + 8g + nvidia display card.. it works well LBSuse:~ # grep -i aperture dmesg.txt Checking aperture... Aperture from AGP @ c0000000 old size 32 MB Aperture from AGP @ c0000000 size 512 MB (APSIZE e00) Node 0: aperture @ c0000000 size 512 MB Node 1: aperture @ c0000000 size 512 MB agpgart-amd64 0000:04:00.0: AGP aperture is 512M @ 0xc0000000 PCI-DMA: Reserving 256MB of IOMMU area in the AGP aperture LBSuse:~ # cat /proc/iomem 00000000-0000ffff : reserved 00010000-0009fbff : System RAM 0009fc00-0009ffff : reserved 000a0000-000bffff : PCI Bus #04 000c0000-000dffff : pnp 00:0d 000e0000-000fffff : reserved 00100000-9bfeffff : System RAM 00200000-0096dc83 : Kernel code 0096dc84-00ea89c7 : Kernel data 00fcd000-011d4ae3 : Kernel bss 9bff0000-9bffefff : ACPI Tables 9bfff000-9bffffff : ACPI Non-volatile Storage 9c000000-9c4fffff : PCI Bus #00 9c300000-9c3fffff : PCI Bus 0000:02 9c500000-fc5fffff : PCI Bus #04 9c500000-bc4fffff : PCI Bus 0000:05 a0000000-afffffff : 0000:05:00.0 c0000000-dfffffff : GART c0000000-dfffffff : aperture fc600000-fc9fffff : PCI Bus #00 fc600000-fc7fffff : PCI Bus 0000:01 fc700000-fc77ffff : 0000:01:0b.0 fc7f8000-fc7fbfff : 0000:01:0c.0 fc7fd000-fc7fdfff : 0000:01:00.0 fc7fd000-fc7fdfff : ohci_hcd fc7fe000-fc7fefff : 0000:01:00.1 fc7fe000-fc7fefff : ohci_hcd fc7ff000-fc7ff7ff : 0000:01:0c.0 fc7ff000-fc7ff7ff : ohci1394 fc7ffc00-fc7fffff : 0000:01:0b.0 fc7ffc00-fc7fffff : sata_sil fc800000-fc8fffff : PCI Bus 0000:02 fc8e0000-fc8effff : 0000:02:09.0 fc8f0000-fc8fffff : 0000:02:09.0 fc8f0000-fc8fffff : tg3 fc9fe000-fc9fefff : IOAPIC 2 fc9fe000-fc9fefff : 0000:00:0b.1 fc9ff000-fc9fffff : IOAPIC 1 fc9ff000-fc9fffff : 0000:00:0a.1 fca00000-febfffff : PCI Bus #04 fca00000-feafffff : PCI Bus 0000:05 fd000000-fdffffff : 0000:05:00.0 feae0000-feafffff : 0000:05:00.0 fec00000-ffffffff : PCI Bus #00 fec00000-fec00fff : IOAPIC 0 fec00000-fec00fff : pnp 00:0c fec01000-fec013ff : HPET 0 fee00000-fee00fff : Local APIC fee00000-fee00fff : pnp 00:0c ff780000-ffffffff : reserved 100000000-1ffffffff : System RAM 200000000-fcffffffff : PCI Bus #00 -+-[0000:04]-+-00.0 Advanced Micro Devices [AMD] AMD-8151 System Controller | \-01.0-[0000:05]----00.0 nVidia Corporation NV34 [GeForce FX 5500] \-[0000:00]-+-06.0-[0000:01]--+-00.0 Advanced Micro Devices [AMD] AMD-8111 USB | +-00.1 Advanced Micro Devices [AMD] AMD-8111 USB | +-0b.0 Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller | \-0c.0 Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) +-07.0 Advanced Micro Devices [AMD] AMD-8111 LPC +-07.1 Advanced Micro Devices [AMD] AMD-8111 IDE +-07.2 Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 +-07.3 Advanced Micro Devices [AMD] AMD-8111 ACPI +-07.5 Advanced Micro Devices [AMD] AMD-8111 AC97 Audio +-0a.0-[0000:02]----09.0 Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet +-0a.1 Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC +-0b.0-[0000:03]-- +-0b.1 Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC +-18.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration +-18.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map +-18.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller +-18.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control +-19.0 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration +-19.1 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map +-19.2 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller \-19.3 Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control only difference that you are using ati card...
Comment 11 Yinghai Lu 2008-10-01 23:32:38 UTC
please try tip/master http://people.redhat.com/mingo/tip.git/readme.txt
Comment 12 Ingo Molnar 2008-10-02 02:13:15 UTC
> OK, it's nailed down to the delta between 2.6.26 and 2.6.26-git1. [...] 2.6.26-git1 is 50515af207d410c9f228380e529c56f43c3de0bd. The v2.6.26..50515af207d410c9f2 range is ~2000 commits, and all the x86 updates are in there. Ingo
Comment 13 Joerg Roedel 2008-10-02 02:20:54 UTC
The GART aperture is not covered by E820 (not marked as reserved or usable). May this be a problem?
Comment 14 Duncan 2008-10-02 03:43:33 UTC
(In reply to comment #10) > just tried tip/master with s2885 + 8g + nvidia display card.. > it works well > LBSuse:~ # cat /proc/iomem > only difference that you are using ati card... Are you using a framebuffer or the standard vgacon? I'm using Radeon framebuffer here, and with vga=ask, then mode 1 (or similar vga mode), on a kernel that works, it seems to switch to framebuffer right about the point at which it's crashing on one that doesn't. To be sure, this isn't really going by log, but by "feel". Should I try with it configured out? And would my /proc/iomem be of help also? Should I post the .27-rc iommu=soft or the old .26 gart version, or both? FWIW, my schedule over the next 24 hours isn't good for working on this that much, but the 24-36 after that, Friday and half of Saturday my time, is better... I know time is compressed due to the release cycle point we're at.
Comment 15 H. Peter Anvin 2008-10-02 08:28:39 UTC
Response to #13: that is normal; the GART aperture is usually identified via a BAR rather than a static resource.
Comment 16 Yinghai Lu 2008-10-02 09:16:27 UTC
yes. only require gart aperture is not overlapped with e820 ram areas.
Comment 17 Yinghai Lu 2008-10-02 21:17:10 UTC
enabld fb, it still works well.
Comment 18 Duncan 2008-10-03 10:36:28 UTC
ARRGG! I had started working on the bisect, but let's just say I brown-paper-bagged something... and I'm glad I have tested-working backups! The bisect data appears to be safe, tho. Here's the log I have so far. I think it was < 100 commits left to test, so that should narrow it down significantly: git bisect start # bad: [50515af207d410c9f228380e529c56f43c3de0bd] firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR git bisect bad 50515af207d410c9f228380e529c56f43c3de0bd # good: [bce7f793daec3e65ec5c5705d2457b81fe7b5725] Linux 2.6.26 git bisect good bce7f793daec3e65ec5c5705d2457b81fe7b5725 # bad: [d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8] Merge branch 'sched/for-linus' into tracing/for-linus git bisect bad d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8 # good: [3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce] Merge branch 'x86/mpparse' into x86/devel git bisect good 3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce # good: [e93ef949fd9a3f237aedfb8e64414b28980530b8] x86: rename paravirtualized TSC functions git bisect good e93ef949fd9a3f237aedfb8e64414b28980530b8 # good: [b6770c83b4b88dd832cc77916b1935b0311c64e0] x86, VisWS: do not allow VisWS for Voyager git bisect good b6770c83b4b88dd832cc77916b1935b0311c64e0 (The paravert TSC, e93ef949f..., was a shot in the dark, but I got another in before the incident to confirm it, so...) I really gotta sleep now before I brown-paper-bag anything else. After that it'll probably take a few hours figuring out and restoring status, before I can get back to this. Hopefully tomorrow.
Comment 19 Ingo Molnar 2008-10-04 01:01:44 UTC
* email@example.com <firstname.lastname@example.org> wrote: > I really gotta sleep now before I brown-paper-bag anything else. > After that it'll probably take a few hours figuring out and restoring > status, before I can get back to this. Hopefully tomorrow. yes, the bisection seems to start to point in the right direction. Suspect commits are: 2387ce5: x86: make 64bit hpet_set_mapping to use ioremap too, v2 87a1c44: x86: get x86_phys_bits early 32b23e9: x86: max_low_pfn_mapped fix #4 7b479be: x86, e820: remove end_user_pfn 9958e81: x86: max_low_pfn_mapped fix, #3 965194c: x86: max_low_pfn_mapped fix, #2 7ab073b: x86: max_low_pfn_mapped fix, #1 69a7704: x86: e820: user-defined memory maps: remove the range instead of update a361ee5: x86: fix /dev/mem compatibility under PAT but any of the other x86 commits could be the culprit too. here's your current bisection log, in a copy&paste-able form: git bisect start git bisect bad 50515af207d410c9f228380e529c56f43c3de0bd git bisect good bce7f793daec3e65ec5c5705d2457b81fe7b5725 git bisect bad d14c8a680ccfdeb5e7b9be4d61162c2b373bd1e8 git bisect good 3de352bbd86f890dd0c5e1c09a6a1b0b29e0f8ce git bisect good e93ef949fd9a3f237aedfb8e64414b28980530b8 git bisect good b6770c83b4b88dd832cc77916b1935b0311c64e0 Ingo
Comment 20 Duncan 2008-10-04 13:40:11 UTC
Created attachment 18164 [details] full bisect log OK, completed the bisect, but is there a way to spit out the result again, without a replay? Here's the log in any case, and a replay spits out the following: f361a450bf1ad14e2b003217dbf3958638631265 is first bad commit commit f361a450bf1ad14e2b003217dbf3958638631265 Author: Yinghai Lu <email@example.com> Date: Thu Jul 10 20:38:26 2008 -0700 x86: introduce max_low_pfn_mapped for 64-bit when more than 4g memory is installed, don't map the big hole below 4g. Signed-off-by: Yinghai Lu <firstname.lastname@example.org> Cc: Suresh Siddha <email@example.com> Signed-off-by: Ingo Molnar <firstname.lastname@example.org> :040000 040000 c8d1071138c28eaf8544f3d956a573cf0ad4bc84 c866895ac7917c7327058e3ae5de677d0eaaacf1 M arch :040000 040000 9bb54b21a753af68a5e6e7ffff4cf18678d179c0 7e39996b3df55d851fabef79de351c0a431cf3cc M include
Comment 21 Anonymous Emailer 2008-10-04 14:56:59 UTC
Reply-To: email@example.com On Sat, Oct 4, 2008 at 1:40 PM, <firstname.lastname@example.org> wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=11676 > > > > > > ------- Comment #20 from email@example.com 2008-10-04 13:40 ------- > Created an attachment (id=18164) > --> (http://bugzilla.kernel.org/attachment.cgi?id=18164&action=view) > full bisect log > > OK, completed the bisect, but is there a way to spit out the result again, > without a replay? Here's the log in any case, and a replay spits out the > following: > > f361a450bf1ad14e2b003217dbf3958638631265 is first bad commit > commit f361a450bf1ad14e2b003217dbf3958638631265 > Author: Yinghai Lu <firstname.lastname@example.org> > Date: Thu Jul 10 20:38:26 2008 -0700 > > x86: introduce max_low_pfn_mapped for 64-bit > > when more than 4g memory is installed, don't map the big hole below 4g. > > Signed-off-by: Yinghai Lu <email@example.com> > Cc: Suresh Siddha <firstname.lastname@example.org> > Signed-off-by: Ingo Molnar <email@example.com> > > :040000 040000 c8d1071138c28eaf8544f3d956a573cf0ad4bc84 > c866895ac7917c7327058e3ae5de677d0eaaacf1 M arch > :040000 040000 9bb54b21a753af68a5e6e7ffff4cf18678d179c0 > 7e39996b3df55d851fabef79de351c0a431cf3cc M include > > could be ati fb want to use direct map address? YH
Comment 22 Yinghai Lu 2008-10-04 15:38:08 UTC
Created attachment 18165 [details] direct map that range when agp is there
Comment 24 Ingo Molnar 2008-10-05 00:23:38 UTC
> could be ati fb want to use direct map address? would be nice to figure out where that happens. Access not via ioremap() is not allowed in general - incidental mapping of some physical memory range in previous kernels might have masked a bug.
Comment 25 Ingo Molnar 2008-10-05 02:21:14 UTC
> please try the patch in comment #22 or try tip/master: http://people.redhat.com/mingo/tip.git/README which has this fix included. Ingo
Comment 26 Duncan 2008-10-05 07:25:03 UTC
The patch is a confirmed fix on linus/master (v2.6.27-rc8-84-gfec6ed1). As for tip, I'll try it, but FWIW here's a bit of feedback. For likely git novices, it may be worth considering whether alternate tree suggestions are worth the potential confusion, or at least get the linked documentation coordinated so it doesn't conflict, thus causing confusion. Up-bug/thread, I was asked to do bisect, then tip/master was suggested. Both suggestions pointed at helpful on-their-own instructions, but the bisect aka git-quick document suggested git-clone, while the tip.git/README doc suggests git remote and etc (yes, I know the tip README isn't for quite the same audience, perhaps a different doc to point testers to might be useful). This was quite confusing and it took me several hours of additional testing time to read up and absorb enough git to resolve the conflicts. I eventually used git remote add to add both, then decided that bisect was the most likely fruitful way to accurately pin-down the problem, since tip has other patches that would have potentially complicated the picture. Resolving this confusion, either by coordinating the docs, or by plainly prioritizing competing suggestions (ie: "Bisect will likely be the most direct route to a fix, but if you aren't comfortable trying that, here's an alternate suggestion."), but preferably both, will make things significantly easier for others taking a path similar to the one I just traveled. The result will be getting test results sooner, or possibly getting them at all, since the cost of confusion may be someone dropping the report entirely, possibly scaring off what might otherwise turn into a regular tester, when they decide it's just too confusing and must be beyond them. Hope it helps...
Comment 27 Ingo Molnar 2008-10-05 08:54:21 UTC
> [...] This was quite confusing and it took me several hours of > additional testing time to read up and absorb enough git to resolve > the conflicts. [...] hm, the -tip readme says: | # | # if you need to do bisection of the -tip tree, then do: | # (but first check that linus/master is indeed a 'good' kernel :-) | # | git bisect start | git bisect good linus/master | git bisect bad tip/master and that is still 100% accurate to bisect bugs that only show up in tip/master. If you want to debug bugs in mainline, you can still use the symbols referenced to in the -tip readme: git bisect start git bisect bad linus/master git bisect good v2.6.24 # last known good kernel So ... it would be very useful if you could suggest a few lines of extra info to tip's readme that would have reduced your confusion. It's meant for git newbies too, so if it causes confusion that's unintentional. Maybe send a patch to that file if you dont mind? Such readmes are hard to write because _we_ are not newbies anymore so we dont notice the hickups in its logic. We very much rely on feedback like yours to improve it. Thanks!
Comment 28 Duncan 2008-10-05 09:01:27 UTC
Confirming tip/master (tip-history-2008-09-09_09.14_Tue-1196-g6ed3e1f) works as well, as expected since it has the patch. Interesting additional config options. I liked the raid-autodetect option, since I turn that off on the kernel command line since I use mdp, and the autodetect loads unpartitioned md instead. Additionally, the built-in kernel command line could be helpful given the length of mine, tho I elected not to use it ATM, since I still consider linus/mainline my usual kernel. It'd be very nice to have both of these in mainline at some point. ... Anyway, now that both tip and mainline-plus-patch are working, I can try some of the other new .27 features, like mtrr cleanup, that I kept off while trying to trace this bug down. =:^)
Comment 29 Duncan 2008-10-05 09:57:14 UTC
(In reply to comment #27) > > [...] This was quite confusing [...] > > hm, the -tip readme says: > > | # > | # if you need to do bisection of the -tip tree [...] First, I'm not sure where all this bug is gated to, but maybe this bit should head elsewhere, another bug or private mail... or doesn't that bother people here (more efficient filtering?) like it can elsewhere? It wasn't the bisect part that was confusing, but setting up the remotes and etc in the first place. git clone is suggested before the bisect stuff in the git-quick doc, as I mentioned, git remote add/update and related commands are suggested in the tip README. That's what confused me. How do they relate to each other and since the tip README was dealing with two trees while git-quick was dealing with one, how does that change things? Further, how does one actually switch between trees? Also, neither document is really clear on remote vs local branches. git-quick assumes one-way-git and non-developer, tip README appears to assume more developer interest, so git-quick would be /expected/ to be simpler in that case... but what are the differences? These are the sorts of questions I was trying to find the answers to. The bisect part was basically fine, GETTING there was the problem. =:^) > Maybe send a patch to that file if you dont mind? Such readmes are hard > to write because _we_ are not newbies anymore so we dont notice the > hickups in its logic. We very much rely on feedback like yours to > improve it. I'm not thinking quite clearly enough on it yet to send a patch... yet. But yes, the above you're not newbies, it's hard to write for them, was where I was headed with this too.
Comment 30 Adrian Bunk 2008-10-07 07:17:55 UTC
fixed by commit d99e90164e6cf2eb85fa94d547d6336f8127a107