Bug 74151

Summary:	PCI core never allocates resource from 64-bit prefetchable space
Product:	Drivers	Reporter:	Guo Chao (yan)
Component:	PCI	Assignee:	drivers_pci (drivers_pci)
Status:	RESOLVED CODE_FIX
Severity:	normal	CC:	benh, bjorn, yinghai
Priority:	P1
Hardware:	PPC-64
OS:	Linux
Kernel Version:	3.14	Subsystem:
Regression:	No	Bisected commit-id:
Attachments:	Dmesg without Yinghai's patch dmesg with Yinghai's patch

Description Guo Chao 2014-04-16 08:21:02 UTC

On power8 machines, we need some 64-bit prefetchable BARs be in 64-bit window. Although we supply such window, PCI core never allocates resource from it which leads to kernel panic, device failure and hindrance to the upcoming SRIOV implementation on this platform.

Here is a solution proposed by Yinghai:

http://patchwork.ozlabs.org/patch/328067/

Comment 1 Guo Chao 2014-04-16 08:23:55 UTC

Created attachment 132451 [details]
Dmesg without Yinghai's patch

Unrelated or sensitive messages are removed or masked

Comment 2 Guo Chao 2014-04-16 08:24:26 UTC

Created attachment 132461 [details]
dmesg with Yinghai's patch

Comment 3 Bjorn Helgaas 2014-05-19 23:49:45 UTC

It would save me a lot of time if you would pull out a specific example of this happening, e.g., we restricted the prefetchable window of bridge X to be below 4GB, even though it is a 64-bit window, because of the 32-bit BAR of device Y.

Comment 4 Guo Chao 2014-05-21 06:32:02 UTC

(In reply to Bjorn Helgaas from comment #3)
> It would save me a lot of time if you would pull out a specific example of
> this happening, e.g., we restricted the prefetchable window of bridge X to
> be below 4GB, even though it is a 64-bit window, because of the 32-bit BAR
> of device Y.

Domain 6 which just consists of a root port and EP will be a simple example. 

("desmg without Yinghai's patch" is copied from console after kernel panic, I don't why many debug information is missing. You may need refer to "dmesg with Yinghai's patch" for window and BAR information)

Domain 6's host bridge resource:

> pci_bus 0006:00: root bus resource [mem 0x3d24800000000-0x3d2487ffeffff] (bus
> address [0x80000000-0xfffeffff])
> pci_bus 0006:00: root bus resource [mem 0x3d24008000000-0x3d247ffffffff 64bit
> pref]
> pci_bus 0006:00: root bus resource [bus 00-ff]

One EP (with 6 functions) is connected to root port. All functions have 64bit pref BARs and ROM BAR. BAR 15 of root port is restricted to be below 4G because of those ROM BARs.

> pci 0006:00:00.0: disabling bridge window [mem 0x00000000-0xffffffffffffffff]
> to [bus 01] (unused)
> pci 0006:00:00.0: BAR 15: assigned [mem 0x3d24800000000-0x3d248017fffff pref]


This is how it looks after Yinghai's fix:

> pci 0006:00:00.0: BAR 15: assigned [mem 0x3d24008000000-0x3d2400fffffff 64bit
> pref]
> pci 0006:00:00.0: BAR 14: assigned [mem 0x3d24800000000-0x3d248007fffff]

Comment 5 Bjorn Helgaas 2014-06-03 23:55:35 UTC

This should be resolved by

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=5b28541552ef5eeffc41d6936105f38c2508e566

which will appear in v3.16-rc1.