Bug 213073
| Summary: | kernel panic early in boot on xeon x5690 dual core | ||
|---|---|---|---|
| Product: | Platform Specific/Hardware | Reporter: | robert.shteynfeld |
| Component: | x86-64 | Assignee: | platform_x86_64 (platform_x86_64) |
| Status: | RESOLVED CODE_FIX | ||
| Severity: | high | CC: | alfalco, bp, dedekind1, mike.rapoport, rppt |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 5.11 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
dmesg after booting 5.10.21 kernel
dmesg booting 5.10.21 with mminit_loglevel=4 ignore_loglevel reoder-memmap-init.patch reoder-memmap-init-v2.patch debug-memmap-init.patch building kernel from stable tree dmesg after booting 5.10.21 kernel with extra debugging reoder-memmap-init-v3.patch 5.12.7 boot with CONFIG_DEBUG_VM=n build reorder-memmap-init-v4.patch 5.12.7 dmesg after patch 4 bdx-CoD-on-v5.13-reverted-dmesg.txt bdx-CoD-off-v5.13-reverted-dmesg.txt bdx-CoD-off-v5.13-vanilla-dmesg.txt config-5.13.0.txt |
||
|
Description
robert.shteynfeld
2021-05-15 02:20:18 UTC
Can you trigger with 5.12 too? Created attachment 296765 [details] attachment-9158-0.html I have not tried myself, but the other user with the same exact problem on same exact hardware did try and it also hangs on boot (from https://bugzilla.redhat.com/show_bug.cgi?id=1945809): "so i update to to the rawhide kernel kernel-5.12.0-0.rc5.180.fc35.x86_64.rpm and the error still exists, I haven't hooked up the laptop to the serial to log the output but it still hangs and kernel panics." On Sat, May 15, 2021 at 3:46 AM <bugzilla-daemon@bugzilla.kernel.org> wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=213073 > > Borislav Petkov (bp@alien8.de) changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > CC| |bp@alien8.de > > --- Comment #1 from Borislav Petkov (bp@alien8.de) --- > Can you trigger with 5.12 too? > > -- > You may reply to this email to add a comment. > > You are receiving this mail because: > You reported the bug. Mike, can you have a look pls? This looks memblock-something related. Thx. Created attachment 296767 [details]
dmesg after booting 5.10.21 kernel
Can you add "mminit_loglevel=4 ignore_loglevel" to the kernel command line and post the log please? I don't have a way to get the console output into a file, but here's the video in case that helps: https://drive.google.com/file/d/1pxIa8e9lP49_ayrPPkqb1EtPVp_U2LnD/view?usp=sharing I've had a early kernel boot problem a few years back that was fixed with a patch and it was due to an unusual memory ranges for the dual x5690. Here's the e-mail chunk from previous problem: Michal Hocko <mhocko@kernel.org> Fri, Jan 25, 2019, 11:39 AM to Linus, Mikhail, Linux, Gerald, Mikhail, Dave, Alexander, Andrew, Pavel, Steven, Daniel, Bob, me On Fri 25-01-19 11:16:30, robert shteynfeld wrote: > Attached is the dmesg from patched kernel. Your Node1 physical memory range precedes Node0 which is quite unusual but it shouldn't be a huge problem on its own. But memory ranges are not aligned to the memory section [ 0.286954] Early memory node ranges [ 0.286955] node 1: [mem 0x0000000000001000-0x0000000000090fff] [ 0.286955] node 1: [mem 0x0000000000100000-0x00000000dbdf8fff] [ 0.286956] node 1: [mem 0x0000000100000000-0x0000001423ffffff] [ 0.286956] node 0: [mem 0x0000001424000000-0x0000002023ffffff] As you can see the last pfn for the node1 is inside the section and Node0 starts right after. This is quite unusual as well. If for no other reasons then the memmap of those struct pages will be remote for one or the other. Actually I am not even sure we can handle that properly because we do expect 1:1 mapping between sections and nodes. Now it also makes some sense why 2830bf6f05fb ("mm, memory_hotplug: initialize struct pages for the full memory section") made any difference. We simply write over a potentially initialized struct page and blow up on that. I strongly suspect that the commit just uncovered a pre-existing problem. Let me think what we can do about that. As per redhat thread, disabling NUMA avoids the problem. Can you run 5.10.y with "mminit_loglevel=4 ignore_loglevel" please? Created attachment 296789 [details]
dmesg booting 5.10.21 with mminit_loglevel=4 ignore_loglevel
Booted 5.10.21 with mminit_loglevel=4 ignore_loglevel as requested.
Created attachment 296813 [details]
reoder-memmap-init.patch
I believe that regression is caused by the refactoring of the memory map for holes in the memory [commit 0740a50b9baa ("mm/page_alloc.c: refactor initialization of .
struct page for holes in memory layout")].
It presumed that the nodes are sorted in ascending order by their start_pfn and this assumption does not hold on your system.
The attached patch enforces that order before starting memory map initialization and I think it should fix your issue.
Thanks for the quick patch! Created attachment 296913 [details]
reoder-memmap-init-v2.patch
Can you please test this version of the patch as well?
It is more efficient than the previous one, especially for large systems
Sorry, I never got to test the first patch. I tried the second one but it doesn't seem to work. I'll try the first on Mon. Tested the first patch, but still getting kernel panic. One thing to note is that it's building 5.12.5 kernel when I build branch origin/f33 of the fedora kernel. Here's the video in case it helps: https://drive.google.com/file/d/1sKwKeKSktIXXbGNzFXCG-7ttXRUhE9Al/view?usp=sharing Created attachment 297009 [details]
debug-memmap-init.patch
This patch adds some debugging prints to v5.10.21.
Can you apply this to v5.10.21 and post the boot log?
Created attachment 297013 [details] attachment-19828-0.html I tried checking out the 5.10.21 commit and running "fedpkg local" to build, but it refuses to do that with a detached HEAD git status. So not exactly sure how I can build an old release. Will try to figure it out. On Thu, May 27, 2021 at 2:08 PM <bugzilla-daemon@bugzilla.kernel.org> wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=213073 > > --- Comment #16 from Mike Rapoport (mike.rapoport@gmail.com) --- > Created attachment 297009 [details] > --> https://bugzilla.kernel.org/attachment.cgi?id=297009&action=edit > debug-memmap-init.patch > > <https://bugzilla.kernel.org/attachment.cgi?id=297009&action=editdebug-memmap-init.patch> > > This patch adds some debugging prints to v5.10.21. > Can you apply this to v5.10.21 and post the boot log? > > -- > You may reply to this email to add a comment. > > You are receiving this mail because: > You reported the bug. Created attachment 297019 [details]
building kernel from stable tree
I think that for this test you may build a kernel without proper Fedora packaging.
Something like the attached script should work.
I built the kernel rpm, but it's 1G and my /boot is a 500MB partition. Is there a reason why the rpm is so large? Hmm, probably it's because of the debug info. I think if you disable it in "Kernel Hacking" -> "Compile-time checks and compiler options" -> "Compile the kernel with debug info" in kernel configuration (e.g. make menuconfig) the size of the rpm will be much smaller. Created attachment 297047 [details]
dmesg after booting 5.10.21 kernel with extra debugging
Created attachment 297057 [details]
reoder-memmap-init-v3.patch
The prints actually seem to be quite ok.
I've updated the second patch to include the same prints, and I've added ~1sec delay inside the initialization of the memory map, so that a video could be more helpful.
I think that it's also possible to disable CONFIG_DEBUG_VM and CONFIG_DEBUG_VM_PGFLAGS in "Kernel hacking" -> "Memory Debugging" to have the system boot to get the logs.
Created attachment 297143 [details]
5.12.7 boot with CONFIG_DEBUG_VM=n build
Was not able to apply the latest patch to any of the versions I've tried, but was able to build with CONFIG_DEBUG_VM off and save the dmesg output from 5.12.7 (does not panic as you suggested).
Created attachment 297155 [details]
reorder-memmap-init-v4.patch
Hmm, something went wrong with reorder-memmap-init-v3.patch, I diffed the wrong versions.
Can you please try v4? It should apply on v5.12.7 and it should boot with CONFIG_DEBUG_VM=n.
Even if it doesn't it has delays so that video will be helpful.
Hi, I have the same behavior reported here, on a Dell Precision T7500 with two Xeon X5675 if it has NUMA enabled in the BIOS. How could I help testing the v4 patch? The patch applies to v5.12.7, if you could apply it, build the kernel and send the dmesg output that would be great. Created attachment 297419 [details]
5.12.7 dmesg after patch 4
The prints seem to be correct and the system booted Ok, didn't it? Was CONFIG_DEBUG_VM enabled or disabled? I thought that CONFIG_DEBUG_VM=y was set from previous build, but it looks like it wasn't set (I'm looking at the .config file?). The system booted ok. From .config # CONFIG_DEBUG_VM is not set # CONFIG_DEBUG_VM_PGTABLE is not set From .config # CONFIG_DEBUG_VM is not set # CONFIG_DEBUG_VM_PGTABLE is not set Can you please verify that with CONFIG_DEBUG_VM=y, CONFIG_DEBUG_VM_PGTABLE=y and CONFIG_PAGE_POISONING=y the system boots fine? Yes it boots. Mike Rappaport referred me to this ticket. Here is what I got. 1. System: Dell R630, 2-Socket Broadwell Xeon 2. What happens: kernel v5.11 boots, kernel v5.12 and v5.13 does not boot. 3. Bisecting result: reverting this commit fixes the problem: 0740a50b9baa (refs/bisect/bad) mm/page_alloc.c: refactor initialization of struct page for holes in memory layout 4. I do not see anything on serial, even with early console. Tried both earlyprintk and earlycon. I used PXE boot for this system with pxelinux booloader. I did not try off-disk boot. 5. The system has the Cluster On Die (CoD) feature enabled. When I disable the CoD feature in BIOS meny, the system boots just fine. In other words, to make the system bood, I need to do one of: a. Revert the patch. Then both CoD on and off boot. b. Keep the patch, but disable CoD. 6. Attached dmesg for the following 3 configurations. a. bdx-CoD-on-v5.13-reverted-dmesg.txt - CoD enabled, kernel is v5.13 + 0740a50b9baa reverted. It reverts cleanly. b. bdx-CoD-off-v5.13-reverted-dmesg.txt - CoD disabled, kernel is v5.13 + 0740a50b9baa reverted. c. bdx-CoD-off-v5.13-vanilla-dmesg.txt - CoD disabled, kernel is vanilla v5.13 7. Attached to kernel config file: config-5.13.0.txt Created attachment 297797 [details]
bdx-CoD-on-v5.13-reverted-dmesg.txt
Created attachment 297799 [details]
bdx-CoD-off-v5.13-reverted-dmesg.txt
Created attachment 297801 [details]
bdx-CoD-off-v5.13-vanilla-dmesg.txt
Created attachment 297803 [details]
config-5.13.0.txt
The issue that was originally reported in this bug is fixed by commit
122e093c1734 ("mm/page_alloc: fix memory map initialization for descending nodes") that went into v5.14-rc1.
Can you please check what happens in your case with 5.14-rc1?
Patch 122e093c1734 fixes the issue for me. How tested: $ git reset --hard v5.13 $ git cherry-pick 122e093c1734 Build, it boots fine. Without 122e093c1734 it does not boot. Thanks! Consider closing the ticket as it is fixed by commit 122e093c1734. The commit has 'Cc: stable@kernel.org', so it eventually should propagate to older stable trees. Ok, closing. |