Bug 94361

The below BUG event happens during PCI bus enumeration on some of my
gear. In particular the Advanced Telecommunications Architecture (ATCA)
has carrier cards that contain Field Replaceable Units (FRUs). FRUs
are all attached by PCI-to-PCI bridges and some may be empty.

So architecturally the main card is just an array of eight bridges
and the CPU/computer is just in one slot.

carrier |--- adapter 1
PCI     |--- (empty)
bus     |--- CPU (fru)
        |--- adapter 4
       ... etc.

The CPU module sees this as a PCI bus with all the normal things
on the local PCI bus within its FRU and then a bridge to a
tree of bridges, and some of those bridges go nowhere.

CPU -|--- memory controller
     |--- whatever
     |--- PCI bridge(#) -|--- PCI bridge -|--- adapter 1 item 1
                         |                |--- adapter 1 item 2
                         |
                         |--- PCI bridge -|--- adapter 4 item 1
                                          |--- adapter 4 item 2

(#)Actually I think there is another layer of bridges in there
but I am running out of ASCII art space.

The longest link is something like
CPU to local bus
local bus to plug bus
plug bus to backplane
backplane to other plug bus
other plug bus to target local bus
target local bus to device.

Anyway, I am taking a system that is working under 2.x where this
bridge to bridge (to bridge?) thing worked and it's bugging out
on 3.x (at least 3.18 and 3.19, I have no knowledge of 3.x for
x less than 18).

I got as far as seeing that its a composite pointer deref thats
going bad in pci_aspm_init_link_state according to gdb

parent = pdev->bus->parent->self->link_state;

but the sequencing dependency (e.g. when "self", "parent"
and "bus" is really set for each item) is making my brain hurt.



[    1.590865] BUG: unable to handle kernel NULL pointer dereference at 0000000000000088
[    1.606588] IP: [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850
[    1.620375] PGD 0 
[    1.624436] Oops: 0000 [#1] PREEMPT SMP 
[    1.632387] Modules linked in:
[    1.638536] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.19.0-gentoo #9
[    1.651590] Hardware name: Kontron B3001/B3001, BIOS 4.6.3 08/07/2012
[    1.664472] task: ffff880116b20000 ti: ffff880116b28000 task.ti: ffff880116b28000
[    1.679436] RIP: 0010:[<ffffffff81550324>]  [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850
[    1.698084] RSP: 0000:ffff880116b2b958  EFLAGS: 00010246
[    1.708707] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8801165aae78
[    1.722978] RDX: ffff8801165aae58 RSI: 0000000000000000 RDI: ffff8801165aaf00
[    1.737250] RBP: ffff880116b2b9c8 R08: 0000000000015b80 R09: ffff8801165aae40
[    1.751520] R10: ffff8801165aae40 R11: 000000000000000f R12: ffff8801165aae40
[    1.765791] R13: ffff8801165e8000 R14: 0000000000000000 R15: ffff88011643fc00
[    1.780063] FS:  0000000000000000(0000) GS:ffff88011bc00000(0000) knlGS:0000000000000000
[    1.796243] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.807738] CR2: 0000000000000088 CR3: 0000000002412000 CR4: 00000000000007f0
[    1.822007] Stack:
[    1.826036]  ffff880116b2b988 ffffffff8153b682 ffff8801165e9000 ffff8801165e9000
[    1.840966]  ffff880117038400 0000000000000000 ffff880116b2b9c8 ffffffff8153b761
[    1.855896]  ffff880116b2b9b8 ffff880117038400 0000000000000001 0000000000000000
[    1.870828] Call Trace:
[    1.875727]  [<ffffffff8153b682>] ? pci_device_add+0x122/0x170
[    1.887392]  [<ffffffff8153b761>] ? pci_scan_single_device+0x91/0xc0
[    1.900099]  [<ffffffff8153b865>] pci_scan_slot+0xd5/0x120
[    1.911071]  [<ffffffff8153ca1d>] pci_scan_child_bus+0x2d/0xd0
[    1.922738]  [<ffffffff8153c733>] pci_scan_bridge+0x383/0x640
[    1.934233]  [<ffffffff8153ca75>] pci_scan_child_bus+0x85/0xd0
[    1.945900]  [<ffffffff8153c733>] pci_scan_bridge+0x383/0x640
[    1.957391]  [<ffffffff8153b724>] ? pci_scan_single_device+0x54/0xc0
[    1.970101]  [<ffffffff8153ca75>] pci_scan_child_bus+0x85/0xd0
[    1.981770]  [<ffffffff81b26357>] pci_acpi_scan_root+0x317/0x520
[    1.993784]  [<ffffffff8158c8a3>] acpi_pci_root_add+0x3c9/0x4db
[    2.005623]  [<ffffffff8158e44e>] ? acpi_pnp_match+0x2c/0xa4
[    2.016943]  [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a
[    2.029303]  [<ffffffff81588f15>] acpi_bus_attach+0xcf/0x1bf
[    2.040621]  [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a
[    2.052985]  [<ffffffff817d1f85>] ? device_attach+0x45/0xb0
[    2.064128]  [<ffffffff81588f8f>] acpi_bus_attach+0x149/0x1bf
[    2.075622]  [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a
[    2.087984]  [<ffffffff817d1f85>] ? device_attach+0x45/0xb0
[    2.099130]  [<ffffffff81588f8f>] acpi_bus_attach+0x149/0x1bf
[    2.110623]  [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a
[    2.122983]  [<ffffffff815890f4>] acpi_bus_scan+0x5c/0x67
[    2.133782]  [<ffffffff825bb7e6>] acpi_scan_init+0x6b/0x1a1
[    2.144929]  [<ffffffff825bb617>] acpi_init+0x251/0x26e
[    2.155379]  [<ffffffff825bb3c6>] ? acpi_sleep_proc_init+0x2a/0x2a
[    2.167741]  [<ffffffff810002d8>] do_one_initcall+0x98/0x1e0
[    2.179063]  [<ffffffff810e6900>] ? parse_args+0x150/0x430
[    2.190036]  [<ffffffff8257907c>] kernel_init_freeable+0x17e/0x20b
[    2.202394]  [<ffffffff81d884f0>] ? rest_init+0x90/0x90
[    2.212846]  [<ffffffff81d884f9>] kernel_init+0x9/0xf0
[    2.223125]  [<ffffffff81d9b4ac>] ret_from_fork+0x7c/0xb0
[    2.233922]  [<ffffffff81d884f0>] ? rest_init+0x90/0x90
[    2.244372] Code: 0f 85 e2 fa ff ff 41 80 4c 24 4a 03 b8 01 00 00 00 41 0f b6 54 24 49 e9 4b fb ff ff 0f 1f 00 49 8b 45 10 48 8b 40 10 48 8b 40 38 <48> 8b 80 88 00 00 00 48 85 c0 0f 
[    2.284338] RIP  [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850
[    2.298296]  RSP <ffff880116b2b958>
[    2.305276] CR2: 0000000000000088
[    2.311913] ---[ end trace 153b3907ad1e19ba ]---


(gdb) list *0xffffffff815502ba
0xffffffff815502ba is in pcie_aspm_init_link_state
(drivers/pci/pcie/aspm.c:530).
525             INIT_LIST_HEAD(&link->children);
526             INIT_LIST_HEAD(&link->link);
527             link->pdev = pdev;
528             if (pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM) {
529                     struct pcie_link_state *parent;
530                     parent = pdev->bus->parent->self->link_state;
531                     if (!parent) {
532                             kfree(link);
533                             return NULL;
534                     }

Created attachment 169401 [details]
Very Verbose lspci output

Created attachment 169411 [details]
lspci tree output

Created attachment 169421 [details]
dmesg output from successful 2.6.32 boot of device

Created attachment 169431 [details]
boot console log of panic in 3.19

Oh, in case it wasn't obvious, I took all the unnecessary devices out of the carrier board to simplify the data. Normally many to most of those bridges go somewhere. 8-)

The text in the first comment is the original mailing list comment. All of the attachments were taken in one-after-another with all the unnecessary bits removed, so they should be consistent with one another.

It seems a bit strange, according to your lspci tree info, 
03:00.0 03:00.1 03:00.2 ... were under the same bus 0x03,
but 03:00.0 is a upstream port, upstream port should not lie the same
bus with downstream port (03:00.1/03:00.2... etc)

 \-[0000:00]-+-00.0
             +-01.0-[01]--+-00.0
             |            +-00.1
             |            +-00.2
             |            \-00.3
             +-1c.0-[02-0a]----00.0-[03-0a]--+-00.0-[04]--
             |                               +-01.0-[05]--
             |                               +-02.0-[06]--
             |                               +-03.0-[07]--
             |                               +-08.0-[08]--
             |                               +-09.0-[09]--
             |                               \-0a.0-[0a]--


03:00.0 Class 0604: Device abcd:2104 (rev bc)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 27
	Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 1
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee0300c  Data: 4189
	Capabilities: [68] Express (v1) Upstream Port, MSI 00

The NULL issue may be caused by following code:

static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev)
{
	struct pcie_link_state *link;

	link = kzalloc(sizeof(*link), GFP_KERNEL);
	if (!link)
		return NULL;
	INIT_LIST_HEAD(&link->sibling);
	INIT_LIST_HEAD(&link->children);
	INIT_LIST_HEAD(&link->link);
	link->pdev = pdev;
	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM) {
		struct pcie_link_state *parent;
		parent = pdev->bus->parent->self->link_state;   //here
		if (!parent) {
			kfree(link);
			return NULL;
		}
		link->parent = parent;
		list_add(&link->link, &parent->children);
	}


In your PCI tree, Root port 00:1c.0 connected to a downstream port 02:00.0,
and the downstream port 02:00.0 connected to multi functions (including upsteam port 03:00.0 and other downstream port), this is a weird PCIe topology.
Normally, root port should connect to a upstream port and the upstream port should have several downstream ports.

Another, why your board run up in 2.6.32, because, in 2.6.32, if aspm_disabled set(in your system, it was set because ACPI issue), it would return when come in
pcie_aspm_init_link_state().

2.6.32 pcie_aspm_init_link_state（） code

561 void pcie_aspm_init_link_state(struct pci_dev *pdev)
562 {
563     struct pcie_link_state *link;
564     int blacklist = !!pcie_aspm_sanity_check(pdev);
565 
566     if (aspm_disabled || !pdev->is_pcie || pdev->link_state)  //return here
567         return;
568     if (pdev->pcie_type != PCI_EXP_TYPE_ROOT_PORT &&
569         pdev->pcie_type != PCI_EXP_TYPE_DOWNSTREAM)
570         return;
571 
572     /* VIA has a 

2.6.32 boot up log:
[    0.775314] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it

Hi Bjorn, do we need to fix this issue which was caused by the weird pci tree?

(In reply to Yijing Wang from comment #8)
> The NULL issue may be caused by following code:

Yep. The line you indicated was what I got from the gdb info.

>               parent = pdev->bus->parent->self->link_state;   //here

> In your PCI tree, Root port 00:1c.0 connected to a downstream port 02:00.0,
> and the downstream port 02:00.0 connected to multi functions (including
> upsteam port 03:00.0 and other downstream port), this is a weird PCIe
> topology.

Yes. It is part of the Advanced Telecommunication Computing Architecture (ATCA) http://en.wikipedia.org/wiki/Advanced_Telecommunications_Computing_Architecture

The deal is that the "Field Replaceable Units" (FRUs) fit each fit into a carrier card that is little than a power control matrix and a PCIe bus. The CPU module itself is just another FRU just like the targets. So the computing module doesn't own the top level bus. The data trip to the final target devices, if they aren't co-resident on the CPU module, is up to the backplane and then back down to the target controller.

So in this setup the CPU is a peer of all the other adapters.

> Normally, root port should connect to a upstream port and the upstream port
> should have several downstream ports.

Modular systems... go figure...

> Another, why your board run up in 2.6.32, because, in 2.6.32, if
> aspm_disabled set(in your system, it was set because ACPI issue), it would
> return when come in
> pcie_aspm_init_link_state().
> 
> 2.6.32 pcie_aspm_init_link_state（） code
> 
> 561 void pcie_aspm_init_link_state(struct pci_dev *pdev)
> 562 {
> 563     struct pcie_link_state *link;
> 564     int blacklist = !!pcie_aspm_sanity_check(pdev);
> 565 
> 566     if (aspm_disabled || !pdev->is_pcie || pdev->link_state)  //return
> here
> 567         return;
> 568     if (pdev->pcie_type != PCI_EXP_TYPE_ROOT_PORT &&
> 569         pdev->pcie_type != PCI_EXP_TYPE_DOWNSTREAM)
> 570         return;
> 571 
> 572     /* VIA has a 
> 
> 2.6.32 boot up log:
> [    0.775314] ACPI FADT declares the system doesn't support PCIe ASPM, so
> disable it

Is there a reason that 3.x would ignore the FADT and try to activate the ASPM?

Would disabling ASPM fix this at least temporarily?

I appreciate that this is something of a one-off at this time, but it is legal by the PCIe standard and ATCA is a growing concern.

Hi Robert, you could try to add "pcie_aspm=off" to boot command, and then OS would not go in pcie_aspm_init_link_state() again.

Created attachment 174271 [details]
A simple patch to fix NULL poninter for ASPM

Robert, this is a simple fix to avoid NULL pointer for ASPM

This should be fixed by this commit:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c8fc9339409d

which appeared in v4.2.  Closing; please reopen if you still see a problem.