Created attachment 169391 [details] BUG oops output
The below BUG event happens during PCI bus enumeration on some of my gear. In particular the Advanced Telecommunications Architecture (ATCA) has carrier cards that contain Field Replaceable Units (FRUs). FRUs are all attached by PCI-to-PCI bridges and some may be empty. So architecturally the main card is just an array of eight bridges and the CPU/computer is just in one slot. carrier |--- adapter 1 PCI |--- (empty) bus |--- CPU (fru) |--- adapter 4 ... etc. The CPU module sees this as a PCI bus with all the normal things on the local PCI bus within its FRU and then a bridge to a tree of bridges, and some of those bridges go nowhere. CPU -|--- memory controller |--- whatever |--- PCI bridge(#) -|--- PCI bridge -|--- adapter 1 item 1 | |--- adapter 1 item 2 | |--- PCI bridge -|--- adapter 4 item 1 |--- adapter 4 item 2 (#)Actually I think there is another layer of bridges in there but I am running out of ASCII art space. The longest link is something like CPU to local bus local bus to plug bus plug bus to backplane backplane to other plug bus other plug bus to target local bus target local bus to device. Anyway, I am taking a system that is working under 2.x where this bridge to bridge (to bridge?) thing worked and it's bugging out on 3.x (at least 3.18 and 3.19, I have no knowledge of 3.x for x less than 18). I got as far as seeing that its a composite pointer deref thats going bad in pci_aspm_init_link_state according to gdb parent = pdev->bus->parent->self->link_state; but the sequencing dependency (e.g. when "self", "parent" and "bus" is really set for each item) is making my brain hurt. [ 1.590865] BUG: unable to handle kernel NULL pointer dereference at 0000000000000088 [ 1.606588] IP: [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850 [ 1.620375] PGD 0 [ 1.624436] Oops: 0000 [#1] PREEMPT SMP [ 1.632387] Modules linked in: [ 1.638536] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.19.0-gentoo #9 [ 1.651590] Hardware name: Kontron B3001/B3001, BIOS 4.6.3 08/07/2012 [ 1.664472] task: ffff880116b20000 ti: ffff880116b28000 task.ti: ffff880116b28000 [ 1.679436] RIP: 0010:[<ffffffff81550324>] [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850 [ 1.698084] RSP: 0000:ffff880116b2b958 EFLAGS: 00010246 [ 1.708707] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8801165aae78 [ 1.722978] RDX: ffff8801165aae58 RSI: 0000000000000000 RDI: ffff8801165aaf00 [ 1.737250] RBP: ffff880116b2b9c8 R08: 0000000000015b80 R09: ffff8801165aae40 [ 1.751520] R10: ffff8801165aae40 R11: 000000000000000f R12: ffff8801165aae40 [ 1.765791] R13: ffff8801165e8000 R14: 0000000000000000 R15: ffff88011643fc00 [ 1.780063] FS: 0000000000000000(0000) GS:ffff88011bc00000(0000) knlGS:0000000000000000 [ 1.796243] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1.807738] CR2: 0000000000000088 CR3: 0000000002412000 CR4: 00000000000007f0 [ 1.822007] Stack: [ 1.826036] ffff880116b2b988 ffffffff8153b682 ffff8801165e9000 ffff8801165e9000 [ 1.840966] ffff880117038400 0000000000000000 ffff880116b2b9c8 ffffffff8153b761 [ 1.855896] ffff880116b2b9b8 ffff880117038400 0000000000000001 0000000000000000 [ 1.870828] Call Trace: [ 1.875727] [<ffffffff8153b682>] ? pci_device_add+0x122/0x170 [ 1.887392] [<ffffffff8153b761>] ? pci_scan_single_device+0x91/0xc0 [ 1.900099] [<ffffffff8153b865>] pci_scan_slot+0xd5/0x120 [ 1.911071] [<ffffffff8153ca1d>] pci_scan_child_bus+0x2d/0xd0 [ 1.922738] [<ffffffff8153c733>] pci_scan_bridge+0x383/0x640 [ 1.934233] [<ffffffff8153ca75>] pci_scan_child_bus+0x85/0xd0 [ 1.945900] [<ffffffff8153c733>] pci_scan_bridge+0x383/0x640 [ 1.957391] [<ffffffff8153b724>] ? pci_scan_single_device+0x54/0xc0 [ 1.970101] [<ffffffff8153ca75>] pci_scan_child_bus+0x85/0xd0 [ 1.981770] [<ffffffff81b26357>] pci_acpi_scan_root+0x317/0x520 [ 1.993784] [<ffffffff8158c8a3>] acpi_pci_root_add+0x3c9/0x4db [ 2.005623] [<ffffffff8158e44e>] ? acpi_pnp_match+0x2c/0xa4 [ 2.016943] [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a [ 2.029303] [<ffffffff81588f15>] acpi_bus_attach+0xcf/0x1bf [ 2.040621] [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a [ 2.052985] [<ffffffff817d1f85>] ? device_attach+0x45/0xb0 [ 2.064128] [<ffffffff81588f8f>] acpi_bus_attach+0x149/0x1bf [ 2.075622] [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a [ 2.087984] [<ffffffff817d1f85>] ? device_attach+0x45/0xb0 [ 2.099130] [<ffffffff81588f8f>] acpi_bus_attach+0x149/0x1bf [ 2.110623] [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a [ 2.122983] [<ffffffff815890f4>] acpi_bus_scan+0x5c/0x67 [ 2.133782] [<ffffffff825bb7e6>] acpi_scan_init+0x6b/0x1a1 [ 2.144929] [<ffffffff825bb617>] acpi_init+0x251/0x26e [ 2.155379] [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a [ 2.167741] [<ffffffff810002d8>] do_one_initcall+0x98/0x1e0 [ 2.179063] [<ffffffff810e6900>] ? parse_args+0x150/0x430 [ 2.190036] [<ffffffff8257907c>] kernel_init_freeable+0x17e/0x20b [ 2.202394] [<ffffffff81d884f0>] ? rest_init+0x90/0x90 [ 2.212846] [<ffffffff81d884f9>] kernel_init+0x9/0xf0 [ 2.223125] [<ffffffff81d9b4ac>] ret_from_fork+0x7c/0xb0 [ 2.233922] [<ffffffff81d884f0>] ? rest_init+0x90/0x90 [ 2.244372] Code: 0f 85 e2 fa ff ff 41 80 4c 24 4a 03 b8 01 00 00 00 41 0f b6 54 24 49 e9 4b fb ff ff 0f 1f 00 49 8b 45 10 48 8b 40 10 48 8b 40 38 <48> 8b 80 88 00 00 00 48 85 c0 0f [ 2.284338] RIP [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850 [ 2.298296] RSP <ffff880116b2b958> [ 2.305276] CR2: 0000000000000088 [ 2.311913] ---[ end trace 153b3907ad1e19ba ]--- (gdb) list *0xffffffff815502ba 0xffffffff815502ba is in pcie_aspm_init_link_state (drivers/pci/pcie/aspm.c:530). 525 INIT_LIST_HEAD(&link->children); 526 INIT_LIST_HEAD(&link->link); 527 link->pdev = pdev; 528 if (pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM) { 529 struct pcie_link_state *parent; 530 parent = pdev->bus->parent->self->link_state; 531 if (!parent) { 532 kfree(link); 533 return NULL; 534 }
Created attachment 169401 [details] Very Verbose lspci output
Created attachment 169411 [details] lspci tree output
Created attachment 169421 [details] dmesg output from successful 2.6.32 boot of device
Created attachment 169431 [details] boot console log of panic in 3.19
Oh, in case it wasn't obvious, I took all the unnecessary devices out of the carrier board to simplify the data. Normally many to most of those bridges go somewhere. 8-) The text in the first comment is the original mailing list comment. All of the attachments were taken in one-after-another with all the unnecessary bits removed, so they should be consistent with one another.
It seems a bit strange, according to your lspci tree info, 03:00.0 03:00.1 03:00.2 ... were under the same bus 0x03, but 03:00.0 is a upstream port, upstream port should not lie the same bus with downstream port (03:00.1/03:00.2... etc) \-[0000:00]-+-00.0 +-01.0-[01]--+-00.0 | +-00.1 | +-00.2 | \-00.3 +-1c.0-[02-0a]----00.0-[03-0a]--+-00.0-[04]-- | +-01.0-[05]-- | +-02.0-[06]-- | +-03.0-[07]-- | +-08.0-[08]-- | +-09.0-[09]-- | \-0a.0-[0a]-- 03:00.0 Class 0604: Device abcd:2104 (rev bc) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 27 Bus: primary=03, secondary=04, subordinate=04, sec-latency=0 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0300c Data: 4189 Capabilities: [68] Express (v1) Upstream Port, MSI 00
The NULL issue may be caused by following code: static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) { struct pcie_link_state *link; link = kzalloc(sizeof(*link), GFP_KERNEL); if (!link) return NULL; INIT_LIST_HEAD(&link->sibling); INIT_LIST_HEAD(&link->children); INIT_LIST_HEAD(&link->link); link->pdev = pdev; if (pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM) { struct pcie_link_state *parent; parent = pdev->bus->parent->self->link_state; //here if (!parent) { kfree(link); return NULL; } link->parent = parent; list_add(&link->link, &parent->children); } In your PCI tree, Root port 00:1c.0 connected to a downstream port 02:00.0, and the downstream port 02:00.0 connected to multi functions (including upsteam port 03:00.0 and other downstream port), this is a weird PCIe topology. Normally, root port should connect to a upstream port and the upstream port should have several downstream ports. Another, why your board run up in 2.6.32, because, in 2.6.32, if aspm_disabled set(in your system, it was set because ACPI issue), it would return when come in pcie_aspm_init_link_state(). 2.6.32 pcie_aspm_init_link_state（） code 561 void pcie_aspm_init_link_state(struct pci_dev *pdev) 562 { 563 struct pcie_link_state *link; 564 int blacklist = !!pcie_aspm_sanity_check(pdev); 565 566 if (aspm_disabled || !pdev->is_pcie || pdev->link_state) //return here 567 return; 568 if (pdev->pcie_type != PCI_EXP_TYPE_ROOT_PORT && 569 pdev->pcie_type != PCI_EXP_TYPE_DOWNSTREAM) 570 return; 571 572 /* VIA has a 2.6.32 boot up log: [ 0.775314] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Hi Bjorn, do we need to fix this issue which was caused by the weird pci tree?
(In reply to Yijing Wang from comment #8) > The NULL issue may be caused by following code: Yep. The line you indicated was what I got from the gdb info. > parent = pdev->bus->parent->self->link_state; //here > In your PCI tree, Root port 00:1c.0 connected to a downstream port 02:00.0, > and the downstream port 02:00.0 connected to multi functions (including > upsteam port 03:00.0 and other downstream port), this is a weird PCIe > topology. Yes. It is part of the Advanced Telecommunication Computing Architecture (ATCA) http://en.wikipedia.org/wiki/Advanced_Telecommunications_Computing_Architecture The deal is that the "Field Replaceable Units" (FRUs) fit each fit into a carrier card that is little than a power control matrix and a PCIe bus. The CPU module itself is just another FRU just like the targets. So the computing module doesn't own the top level bus. The data trip to the final target devices, if they aren't co-resident on the CPU module, is up to the backplane and then back down to the target controller. So in this setup the CPU is a peer of all the other adapters. > Normally, root port should connect to a upstream port and the upstream port > should have several downstream ports. Modular systems... go figure... > Another, why your board run up in 2.6.32, because, in 2.6.32, if > aspm_disabled set(in your system, it was set because ACPI issue), it would > return when come in > pcie_aspm_init_link_state(). > > 2.6.32 pcie_aspm_init_link_state（） code > > 561 void pcie_aspm_init_link_state(struct pci_dev *pdev) > 562 { > 563 struct pcie_link_state *link; > 564 int blacklist = !!pcie_aspm_sanity_check(pdev); > 565 > 566 if (aspm_disabled || !pdev->is_pcie || pdev->link_state) //return > here > 567 return; > 568 if (pdev->pcie_type != PCI_EXP_TYPE_ROOT_PORT && > 569 pdev->pcie_type != PCI_EXP_TYPE_DOWNSTREAM) > 570 return; > 571 > 572 /* VIA has a > > 2.6.32 boot up log: > [ 0.775314] ACPI FADT declares the system doesn't support PCIe ASPM, so > disable it Is there a reason that 3.x would ignore the FADT and try to activate the ASPM? Would disabling ASPM fix this at least temporarily? I appreciate that this is something of a one-off at this time, but it is legal by the PCIe standard and ATCA is a growing concern.
Hi Robert, you could try to add "pcie_aspm=off" to boot command, and then OS would not go in pcie_aspm_init_link_state() again.
Created attachment 174271 [details] A simple patch to fix NULL poninter for ASPM Robert, this is a simple fix to avoid NULL pointer for ASPM
This should be fixed by this commit: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c8fc9339409d which appeared in v4.2. Closing; please reopen if you still see a problem.