Bug 199779 - [Hardware Error] kernel panic upon reboot on HP DL360 Gen9
Summary: [Hardware Error] kernel panic upon reboot on HP DL360 Gen9
Status: RESOLVED CODE_FIX
Alias: None
Product: Drivers
Classification: Unclassified
Component: PCI (show other bugs)
Hardware: i386 Linux
: P1 normal
Assignee: drivers_pci@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-21 07:03 UTC by Ryan Finnie
Modified: 2018-06-19 18:20 UTC (History)
2 users (show)

See Also:
Kernel Version: 4.15-rc1
Subsystem:
Regression: Yes
Bisected commit-id:


Attachments
lspci -vv (147.95 KB, text/plain)
2018-05-21 07:03 UTC, Ryan Finnie
Details
4.17.0-rc5-next-20180517 dmesg (83.15 KB, text/plain)
2018-05-21 19:31 UTC, Ryan Finnie
Details
pcie_pme_remove removed, crash (20.12 KB, text/plain)
2018-05-21 21:05 UTC, Ryan Finnie
Details
lspci -t (2.35 KB, text/plain)
2018-05-21 21:42 UTC, Ryan Finnie
Details
debug_patch.patch (1.64 KB, patch)
2018-05-21 22:03 UTC, Sinan Kaya
Details | Diff
debug patch output (9.63 KB, text/plain)
2018-05-22 01:25 UTC, Ryan Finnie
Details
shutdown log (13.20 KB, text/plain)
2018-05-22 03:42 UTC, Ryan Finnie
Details

Description Ryan Finnie 2018-05-21 07:03:07 UTC
Created attachment 276079 [details]
lspci -vv

On HPe DL360 Gen9 (and possibly other gens and/or products; I haven't been able to test other HP hardware right now, but I do have several DL360 Gen9s I've confirmed on), upon shutdown/reboot, it will crash with:

[  122.447111] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
[  122.447112] {1}[Hardware Error]: event severity: fatal
[  122.447113] {1}[Hardware Error]:  Error 0, type: fatal
[  122.447114] {1}[Hardware Error]:   section_type: PCIe error
[  122.447115] {1}[Hardware Error]:   port_type: 4, root port
[  122.447116] {1}[Hardware Error]:   version: 1.16
[  122.447118] {1}[Hardware Error]:   command: 0x6010, status: 0x0143
[  122.447119] {1}[Hardware Error]:   device_id: 0000:00:01.0
[  122.447119] {1}[Hardware Error]:   slot: 0
[  122.447120] {1}[Hardware Error]:   secondary_bus: 0x03
[  122.447120] {1}[Hardware Error]:   vendor_id: 0x8086, device_id: 0x2f02
[  122.447121] {1}[Hardware Error]:   class_code: 040600
[  122.447122] {1}[Hardware Error]:   bridge: secondary_status: 0x2000, control: 0x0003
[  122.447123] {1}[Hardware Error]:  Error 1, type: fatal
[  122.447123] {1}[Hardware Error]:   section_type: PCIe error
[  122.447124] {1}[Hardware Error]:   port_type: 4, root port
[  122.447125] {1}[Hardware Error]:   version: 1.16
[  122.447125] {1}[Hardware Error]:   command: 0x6010, status: 0x0143
[  122.447126] {1}[Hardware Error]:   device_id: 0000:00:01.0
[  122.447127] {1}[Hardware Error]:   slot: 0
[  122.447127] {1}[Hardware Error]:   secondary_bus: 0x03
[  122.447128] {1}[Hardware Error]:   vendor_id: 0x8086, device_id: 0x2f02
[  122.447129] {1}[Hardware Error]:   class_code: 040600
[  122.447130] {1}[Hardware Error]:   bridge: secondary_status: 0x2000, control: 0x0003
[  122.447131] Kernel panic - not syncing: Fatal hardware error!
[  122.447166] Kernel Offset: 0x1c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  122.459295] ERST: [Firmware Warn]: Firmware does not respond in time.

And after that, upon POST, the storage controller is not happy but does eventually work:

Embedded RAID 1 : Smart Array P440ar Controller - (2048 MB, V6.30) 7 Logical
Drive(s) - Operation Failed
 - 1719-Slot 0 Drive Array - A controller failure event occurred prior
   to this power-up.  (Previous lock up code = 0x13) Action: Install the
   latest controller firmware. If the problem persists, replace the
   controller.

Up to date firmware (P89 01/22/2018, controller 6.30).  Interestingly, on older (circa 2016 but I don't have an exact version) firmware, this manifested as a crash loop:

[529151.035267] NMI: IOCK error (debug interrupt?) for reason 75 on CPU 0.
[529153.222883] Uhhuh. NMI received for unknown reason 25 on CPU 0.
[529153.222884] Do you have a strange power saving mode enabled?
[529153.222884] Dazed and confused, but trying to continue
[529153.554447] Uhhuh. NMI received for unknown reason 25 on CPU 0.
[529153.554448] Do you have a strange power saving mode enabled?
[529153.554449] Dazed and confused, but trying to continue

I've narrowed it down to https://patchwork.kernel.org/patch/10027157/ as part of commit 1b6115fbe3b3db746d7baa11399dd617fc75e1c4; removing that line prevents the panic.
Comment 1 Sinan Kaya 2018-05-21 11:11:51 UTC
Can you test this patch?

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/pci/hotplug?id=d22b362184553899f7d6b6760899a77d3b2d7c1b

There is a known Intel errata that we missed.
Comment 2 Sinan Kaya 2018-05-21 11:24:20 UTC
can you also share your dmesg?
Comment 3 Ryan Finnie 2018-05-21 19:31:17 UTC
Created attachment 276103 [details]
4.17.0-rc5-next-20180517 dmesg
Comment 4 Ryan Finnie 2018-05-21 19:32:26 UTC
Thanks, but same problem with that patch against 4.15.  Even tried next-20180517 to be sure, no luck.  dmesg against next-20180517 has been attached.
Comment 5 Sinan Kaya 2018-05-21 19:38:19 UTC
Cool, I had my suspicions. That's why, I asked for dmesg. Your system doesn't seem to have hotplug driver loaded. The bugfix above is valid only if you have hotplug driver enabled. Something else must be happening.
Comment 6 Sinan Kaya 2018-05-21 19:52:07 UTC
it looks like PME is the only PCIe port service driver loaded. Can you empty out this line to see if it makes any difference? Then, we can start going deeper based on your test result.

https://elixir.bootlin.com/linux/latest/ident/pcie_pme_remove
Comment 7 Ryan Finnie 2018-05-21 21:05:30 UTC
Created attachment 276111 [details]
pcie_pme_remove removed, crash
Comment 8 Ryan Finnie 2018-05-21 21:06:16 UTC
-       .remove         = pcie_pme_remove,

With that removed, the crash becomes:

[  115.008578] kernel BUG at drivers/pci/msi.c:352!
[  115.069730] invalid opcode: 0000 [#1] SMP PTI
[  115.127399] CPU: 15 PID: 1 Comm: systemd-shutdow Not tainted 4.17.0-rc5-next-20180517-custom #1
[  115.242735] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 01/22/2018
[  115.351050] RIP: 0010:free_msi_irqs+0x17b/0x1b0
[  115.410250] Code: 84 e1 fe ff ff 45 31 f6 eb 11 41 83 c6 01 44 39 73 14 0f 86 ce fe ff ff 8b 7b 10 44 01 f7 e8 7c f4 bb ff 48 83 78 70 00 74 e0 <0f> 0b 49 8d b5 a0 00 00 00 e8 b7 a0 bc ff e9 cf fe ff ff 48 8b 78 
[...]

Full output attached.
Comment 9 Sinan Kaya 2018-05-21 21:08:06 UTC
Oops. can you comment out this line only?

https://elixir.bootlin.com/linux/latest/source/drivers/pci/pcie/pme.c#L431

We have to call free_irq(). I went too aggressive at the problem.
Comment 10 Ryan Finnie 2018-05-21 21:30:49 UTC
Commented out "pcie_pme_suspend(srv);", back to original Hardware Error crash.
Comment 11 Sinan Kaya 2018-05-21 21:34:55 UTC
Weird. I'll come up with a debug patch. Can you collect some more data as to what other systems see this issue in the meantime?

Since you are the first one to report the problem, there must be something unique about your setup. 

Also, please attach sudo lspci -t output too.
Comment 12 Ryan Finnie 2018-05-21 21:42:02 UTC
Sure.  I'm seeing this on a set of 4 DL360 Gen9s, I believe they were all purchased at the same time around 2016.  I'll look around for further machines I can test on, looking for:

1) DL360 Gen9s but not from the same batch as these
2) Previous gens (not sure we have any older ones)
3) DL380 Gen9

Attaching lspci -t.
Comment 13 Ryan Finnie 2018-05-21 21:42:24 UTC
Created attachment 276113 [details]
lspci -t
Comment 14 Sinan Kaya 2018-05-21 22:03:33 UTC
Created attachment 276115 [details]
debug_patch.patch
Comment 15 Ryan Finnie 2018-05-22 01:24:49 UTC
I was able to test on another DL360 Gen9 received about a year after the ones I discovered on, same problem.  And a DL380 Gen9 with similar specs, also crashes.  I was able to test on a DL380 Gen10, which did *not* crash.  In summary:

Bad: DL360 Gen9 - BIOS P89 v2.56 (01/22/2018) - P440ar V6.30 (originals)
Bad: DL360 Gen9 - BIOS P89 v2.52 (10/25/2017) - P440ar V6.06 (newer)
Bad: DL380 Gen9 - BIOS P89 v2.52 (10/25/2017) - P440ar V6.06 (newer)
Good: DL380 Gen10 - U30 v1.32 (02/01/2018) - P408i-a 1.04-0 (even newer)

Attached is the output from your debug patch on the original test system.
Comment 16 Ryan Finnie 2018-05-22 01:25:25 UTC
Created attachment 276121 [details]
debug patch output
Comment 17 Sinan Kaya 2018-05-22 01:43:38 UTC
Many thanks, let's try these tests. Debug prints are not giving me any clues. The error seems to be asynchronous to the code execution. We'll have to find out by trial and error which one is confusing the HW. My bet is on the first one followed by the third.

1. comment out this line only.

https://elixir.bootlin.com/linux/v4.17-rc6/source/drivers/pci/pcie/portdrv_core.c#L412

2. Comment out this line only.

https://elixir.bootlin.com/linux/v4.17-rc6/source/drivers/pci/pcie/portdrv_pci.c#L148

3. Comment out the if block only. 

https://elixir.bootlin.com/linux/v4.17-rc6/source/drivers/pci/pcie/portdrv_pci.c#L142
Comment 18 Ryan Finnie 2018-05-22 02:24:18 UTC
Progress!  #1 reboots correctly.

A) I had reverted out the debug print patch, want me to add it back?  Does it give you any extra insight?
B) Should I move on to #2 and #3?
Comment 19 Sinan Kaya 2018-05-22 02:34:51 UTC
No, this is enough. We now understand that disabling the bus master bit in the command-control register of the root port is causing a crash on your system. 

I suspect that the firmware is talking to the PCIe bus in parallel and by disabling the bus master bit, we are breaking the FW.
Comment 20 Sinan Kaya 2018-05-22 02:39:28 UTC
Can you also attach the messages you are seeing during shutdown/reboot? The driver clean up order could be important too.
Comment 21 Ryan Finnie 2018-05-22 03:42:25 UTC
Created attachment 276123 [details]
shutdown log

[`dmesg -n debug` added, otherwise normal systemd-obfuscated user messages]

This particular test machine is a MAAS server, 4 interfaces, 2 bonds, 2 bridges.  It normally runs a KVM instance directly, but I don't have it set up to autoboot to save time while testing.

Functionally, the other machines tested don't have a common operational trait: OpenStack "smoosh" (nova-compute + n-c-c + neutron + swift + ceph etc in LXDs), a straight Apache archive server, a standby firewall.  Actually, they all appear to be at least partially utilizing 10gige interfaces (hopefully that's not a consideration since I'm not sure if I can pull a straight gigabit machine out of active use to test on short notice).
Comment 22 Sinan Kaya 2018-05-22 03:56:49 UTC
can you apply debug_patch.patch + 

1. comment out this line only.

https://elixir.bootlin.com/linux/v4.17-rc6/source/drivers/pci/pcie/portdrv_core.c#L412

and collect shutdown log one more time. 

I see quite a bit of driver shutdown activity from your network adapters. I want to see them in reference to the port service driver shutdown to see which one is happening first and last.
Comment 23 Bjorn Helgaas 2018-05-22 17:14:03 UTC
I am not yet convinced that it is necessary for pcie_port_device_remove() to call pci_disable_device() on PCIe Root Ports and Switch Ports during a reboot.

A similar question came during discussion of pciehp timeouts during shutdown [1].  Eric Biederman had a good response [2] that I haven't had time to assimilate yet.

[1] https://lkml.kernel.org/r/8770820b-85a0-172b-7230-3a44524e6c9f@molgen.mpg.de
[2] https://lkml.kernel.org/r/87tvrrypkv.fsf@xmission.com
Comment 24 Sinan Kaya 2018-05-22 17:33:13 UTC
I think the motivation is for rogue transactions from the devices not to hit the system memory while a new kernel is booted via kexec.

It is not an issue when IOMMU is not present since the second kernel that is booting doesn't share the same address space. 

However; when IOMMU is present, an adapter can corrupt the newly booting kernel. So, you ideally want to have bus master bit cleared for a clean boot. 

What is interesting is that kexec is already doing this job in pci_device_shutdown(). This extra clear is unnecessary. I'll post a patch to remove it.
Comment 25 Sinan Kaya 2018-06-11 18:28:26 UTC
change merged to the 4.18 kernel:

https://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git/commit/?id=0d98ba8d70b0070ac117452ea0b663e26bbf46bf

This issue can be closed.
Comment 26 Ryan Finnie 2018-06-19 18:20:40 UTC
Ack, thank you for all your help.

Note You need to log in before you can comment on or make changes to this bug.