Created attachment 293803 [details] Output of `journalctl -k` On the Dell XPS 13 9310, Linux 5.9.9 logged the messages below. What should the user do with them? ``` kernel: BERT: Error records from previous boot: kernel: [Hardware Error]: event severity: fatal kernel: [Hardware Error]: Error 0, type: fatal kernel: [Hardware Error]: section_type: Firmware Error Record Reference kernel: [Hardware Error]: Firmware Error Record Type: SOC Firmware Error Record Type2 kernel: [Hardware Error]: Revision: 2 kernel: [Hardware Error]: Record Identifier: 8f87f311-c998-4d9e-a0c4-6065518c4f6d kernel: [Hardware Error]: 00000000: 0100a306 00000280 ca5824d3 000003ab .........$X..... kernel: [Hardware Error]: 00000010: 00180000 07840010 800003ff 0001f668 ............h... kernel: [Hardware Error]: 00000020: 0002f5d2 000231d6 00000000 00020000 .....1.......... kernel: [Hardware Error]: 00000030: 7f04c700 f08fff6f 6063ff09 6020f10a ....o.....c`.. ` kernel: [Hardware Error]: 00000040: 1100009f 00000020 01004000 00240000 .... ....@....$. kernel: [Hardware Error]: 00000050: 00000004 00502881 000ecf02 0300035d .....(P.....]... kernel: [Hardware Error]: 00000060: 00000041 70321070 20003c5a 00000e00 A...p.2pZ<. .... kernel: [Hardware Error]: 00000070: c88001fc fc460007 49063fc7 ffff476e ......F..?.InG.. kernel: [Hardware Error]: 00000080: 00021b81 00320044 03416809 3802011e ....D.2..hA....8 kernel: [Hardware Error]: 00000090: 00003870 00060000 00000000 000000c0 p8.............. kernel: [Hardware Error]: 000000a0: 00020000 00000000 3e000000 00fcde70 ...........>p... kernel: [Hardware Error]: 000000b0: 100e0020 00000100 0083c1c0 39c00003 ..............9 kernel: [Hardware Error]: 000000c0: 000000f0 10224000 80005080 c0000006 .....@"..P...... kernel: [Hardware Error]: 000000d0: 00000000 00001080 00034000 020014b8 .........@...... kernel: [Hardware Error]: 000000e0: 02002080 7f05be03 00000001 00000000 . .............. kernel: [Hardware Error]: 000000f0: 07000100 a31b0000 200000a0 00203100 ........... .1 . kernel: [Hardware Error]: 00000100: 00602010 00000000 00000009 01000080 . `............. kernel: [Hardware Error]: 00000110: 457c2080 e077c0fc fffffc7d 00000000 . |E..w.}....... kernel: [Hardware Error]: 00000120: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000130: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000140: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000150: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000160: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000170: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000180: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000190: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000001f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000200: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000210: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000220: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000230: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000240: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000250: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000260: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000270: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000280: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000290: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000002f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000300: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000310: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000320: 00000000 01002f20 00000000 00000000 .... /.......... kernel: [Hardware Error]: 00000330: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000340: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000350: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000360: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000370: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000380: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000390: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000003f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000400: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000410: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000420: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000430: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000440: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000450: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000460: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000470: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000480: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000490: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000004f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000500: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000510: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000520: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000530: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000540: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000550: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000560: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000570: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000580: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000590: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000005f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000600: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000610: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000620: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000630: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000640: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000650: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000660: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000670: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000680: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000690: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000006f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000700: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000710: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000720: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000730: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000740: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000750: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000760: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000770: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000780: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000790: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000007f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000800: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000810: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000820: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000830: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000840: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000850: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000860: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000870: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000880: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000890: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000008f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000900: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000910: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000920: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000930: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000940: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000950: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000960: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000970: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000980: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 00000990: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009a0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009b0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009c0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009d0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009e0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: 000009f0: 00000000 00000000 00000000 00000000 ................ kernel: [Hardware Error]: Error 1, type: fatal kernel: [Hardware Error]: section_type: Firmware Error Record Reference kernel: [Hardware Error]: Firmware Error Record Type: SOC Firmware Error Record Type2 kernel: [Hardware Error]: Revision: 2 kernel: [Hardware Error]: Record Identifier: 8f87f311-c998-4d9e-a0c4-6065518c4f6d kernel: [Hardware Error]: 00000000: 1100a101 00000080 00000000 fe013d20 ............ =.. kernel: [Hardware Error]: 00000010: 00000000 0000260f 98bb732a 0000260e .....&..*s...&.. kernel: [Hardware Error]: 00000020: 98bb742a 0000260f 98bb7516 00002610 *t...&...u...&.. kernel: [Hardware Error]: 00000030: 98bb7626 0000260e 98bb7819 0000260f &v...&...x...&.. kernel: [Hardware Error]: 00000040: 98bb7918 0000260e 98bb7a18 0000260f .y...&...z...&.. kernel: [Hardware Error]: 00000050: 98bb7b1e 00002610 98bb7c2e 00002614 .{...&...|...&.. kernel: [Hardware Error]: 00000060: 98bb7ef1 00002615 98bb8119 00000332 .~...&......2... kernel: [Hardware Error]: 00000070: 00001033 02002616 98bb82ab 0000261a 3....&.......&.. kernel: [Hardware Error]: 00000080: 98bb8443 0000261b 98bb967f 00000332 C....&......2... kernel: [Hardware Error]: 00000090: 00001033 0200261c 98bb97e2 00000000 3....&.......... kernel: [Hardware Error]: 000000a0: 00000001 02002622 98bb9a8a 00000001 ...."&.......... kernel: [Hardware Error]: 000000b0: 00000000 02002623 98bbc766 00000332 ....#&..f...2... kernel: [Hardware Error]: 000000c0: 00001033 0200261d 98bbc9c1 00000332 3....&......2... kernel: [Hardware Error]: 000000d0: 00001033 02002617 98bbcb33 0000260e 3....&..3....&.. kernel: [Hardware Error]: 000000e0: 98bbccb3 0000260f 98bbcd9f 0000260e .....&.......&.. kernel: [Hardware Error]: 000000f0: 98bbce9f 0000260f 98bbcf8b 00002610 .....&.......&.. kernel: [Hardware Error]: 00000100: 98bbd09b 00002607 98bbd273 00000002 .....&..s....... kernel: [Hardware Error]: 00000110: 01002000 55c6316f 00002014 55c632bb . ..o1.U. ...2.U kernel: [Hardware Error]: 00000120: 02002617 98a663fa 0000260e 98a6657a .&...c...&..ze.. kernel: [Hardware Error]: 00000130: 0000260f 98a66666 0000260e 98a66766 .&..ff...&..fg.. kernel: [Hardware Error]: 00000140: 0000260f 98a66852 00002610 98a66962 .&..Rh...&..bi.. kernel: [Hardware Error]: 00000150: 00002607 98a66b3a 00002604 98ae685a .&..:k...&..Zh.. kernel: [Hardware Error]: 00000160: 00002608 98ae6fac 00002609 98ae709e .&...o...&...p.. kernel: [Hardware Error]: 00000170: 0000260a 98ae71b2 0000260b 98b0b418 .&...q...&...... kernel: [Hardware Error]: 00000180: 0000260c 98b0b51a 0000260d 98b35853 .&.......&..SX.. kernel: [Hardware Error]: 00000190: 00002608 98b359cb 00002609 98b35abd .&...Y...&...Z.. kernel: [Hardware Error]: 000001a0: 0000260a 98b35bd1 0000260b 98b71704 .&...[...&...... kernel: [Hardware Error]: 000001b0: 0000260c 98b71806 0000260d 98bb6389 .&.......&...c.. kernel: [Hardware Error]: 000001c0: 00002605 98bb667e 00002606 98bb6b4a .&..~f...&..Jk.. kernel: [Hardware Error]: 000001d0: 0000260e 98bb6c57 0000260f 98bb6d49 .&..Wl...&..Im.. kernel: [Hardware Error]: 000001e0: 0000260e 98bb6e49 0000260f 98bb6f3b .&..In...&..;o.. kernel: [Hardware Error]: 000001f0: 00002610 98bb704b 0000260e 98bb723e .&..Kp...&..>r.. ```
Is this from the bad TGL microcode? See https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/44
(In reply to Mario Limonciello from comment #1) > Is this from the bad TGL microcode? See > https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/44 As only the Debian packages are used, I do not think so, as the TGL-UP3 μ-code update was removed from the package [1] due to the referenced bug. Is there a way to decode the BERT error? What does it complain about? [1]: https://salsa.debian.org/hmh/intel-microcode/-/commit/46f79d4bbab5ce09d9b4aa15841c020f65a36db6
Debian packages had the ucode in there though and it's complaining about a previous boot - no? Regarding decoding the BERT error, you can see the ACPI spec for details (it's in as far back as 6.0 - we're currently at 6.3): https://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf
Paul, any update? It is just you're lucky to have a firmware that tells you extra information of the previous failure. So if you often got this BERT log after system failures, then let's fix the failure first. If you got this BERT log very occasionally, then you can ignore it, just like you can ignore the failure that causes this log. :) what do you think?
I do not have access to the system anymore, so can’t do anymore tests.
I have the same/a similar issue on a Clevo NH77HPQ laptop (with Intel i7-11800H) running Fedora 35. The crash is preceded by screen corruption (looks like https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/44). If the bug is distribution-specific, then I understand it can't be solved here. I believe the BERT data comes through the new CrashLog feature from the PMC/PCH and CPU, so the ACPI spec doesn't help decode the hexdump. I actually don't know if users have a way to decode CrashLog dumps. I've ruled out the dGPU: the issue appears with regardless of nouveau, proprietary NVIDIA, or nouveau with "nouveau.modeset=0". It's also unrelated to i915 GuC mode or default, and microcode 0x2C or 0x3C. I've also tried disabling PCIe ASPM and L1 substates in UEFI. "split_lock_detect=off" is also unrelated; I know that this is not great debugging methodology, but I can't parse the CrashLog. Can I re-open this bug, or should I open another?
Wanted to add that I am experiencing the same crash on my Proxmox (Debian Bullseye) server. I'm running it on an Intel NUC 11PAH (with an i7-1165G7 and intel microcode update 0x88). The crash happens randomly while idling, with intermittent frequency. I had it stay on for over a day the first time I set it up, but it's been unable to stay stable for longer than an couple of hours since then. It also is random as to whether it recovers and reboots on its own or if it just hangs and requires a manual hard reboot.
(In reply to Santiago Loane from comment #7) > Wanted to add that I am experiencing the same crash on my Proxmox (Debian > Bullseye) server. I'm running it on an Intel NUC 11PAH (with an i7-1165G7 > and intel microcode update 0x88). > > The crash happens randomly while idling, with intermittent frequency. I had > it stay on for over a day the first time I set it up, but it's been unable > to stay stable for longer than an couple of hours since then. It also is > random as to whether it recovers and reboots on its own or if it just hangs > and requires a manual hard reboot. Do you have already found a fix for your issue? I have exactly the same issue with a brand new nuc running the latest version of Proxmox