Bug 88131

Summary: PCI device intel ixgbe and xhci with 64-bit bar address can NOT work on x86_32 PAE kernel.
Product: Drivers Reporter: Pengyu Ma (mapengyu)
Component: PCIAssignee: drivers_pci (drivers_pci)
Status: RESOLVED CODE_FIX    
Severity: normal CC: bjorn, mapengyu, yinghai
Priority: P1    
Hardware: i386   
OS: Linux   
Kernel Version: 3.18-rc4 Subsystem:
Regression: No Bisected commit-id:
Attachments: dmesg
PCI: Don't reject 64bit mmio on 32bit/PAE mode
RFC: x86, PCI: support mmio more than 44 bits on 32bit/PAE mode
ixgbe and xhci failed after applied 2 patches
v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode
boot OK with v2 patches
patch that force reassign that could cause xhci hang
3.18-rc4 with 3 patches boot hang
disable_usb_legacy_early_2014_11_19 with 2 patches boot ok
disable_usb_legacy_early_2014_11_19 with 3 patches boot hang

Description Pengyu Ma 2014-11-13 03:57:58 UTC
Created attachment 157441 [details]
dmesg

PCI device intel ixgbe and xhci with 64-bit bar address can NOT work on x86_32 kernel.

The bar0 base address of intel ixgbe ethernet card and xhci are above 4G.
The commit d56dbf5bab8ce44c5407bb099f71987f58d18bb4 try to map the resource above 4G.
Enabled PAE and HIGHMEM64G, it should support 64-bit BAR, but failed on high bar address.
Comment 1 Yinghai Lu 2014-11-13 08:27:21 UTC
Created attachment 157461 [details]
PCI: Don't reject 64bit mmio on 32bit/PAE mode
Comment 2 Yinghai Lu 2014-11-13 08:28:25 UTC
Created attachment 157471 [details]
RFC: x86, PCI: support mmio more than 44 bits on 32bit/PAE mode
Comment 3 Yinghai Lu 2014-11-13 08:29:23 UTC
Please try the two patches at the same time.
Comment 4 Pengyu Ma 2014-11-13 09:21:22 UTC
Created attachment 157481 [details]
ixgbe and xhci failed after applied 2 patches

load ixgbe and xhci failed after applied 2 patches.
Please check the dmesg.

Thanks.
Comment 5 Yinghai Lu 2014-11-13 23:02:11 UTC
Created attachment 157491 [details]
v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode

more simple version to fix ioremap on 32bit/PAE on MMIO above 44bits
Comment 6 Pengyu Ma 2014-11-14 02:30:29 UTC
v2 patch with the pci patch are working well now.
Thanks for Yinghai's great help.

I think you will send these patches to ml kernel.
When will they can be integrated?
Comment 7 Yinghai Lu 2014-11-14 03:10:23 UTC
Can you attached boot log after two patches?
Comment 8 Pengyu Ma 2014-11-14 05:17:20 UTC
Created attachment 157511 [details]
boot OK with v2 patches

Thers is a error log after hours running:
"UDP: bad checksum. From 128.224.179.59:1534 to 128.224.179.255:1534 ulen 261"
Comment 9 Yinghai Lu 2014-11-14 20:43:50 UTC
Created attachment 157651 [details]
patch that force reassign that could cause xhci hang

for completeness...here is that previous patch that cause xhci hang and it is
replaced by "x86, PCI: support mmio more than 44 bit on 32bit/PAE mode"
Comment 10 Pengyu Ma 2014-11-19 06:45:50 UTC
Created attachment 158171 [details]
3.18-rc4 with 3 patches boot hang

patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch2: v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode
patch3: patch that force reassign that could cause xhci hang 

With above 3 patches to kernel 3.18-rc4
boot hang
Comment 11 Yinghai Lu 2014-11-19 07:21:00 UTC
How about only with 

patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch2: v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode

?
Comment 12 Pengyu Ma 2014-11-19 07:40:14 UTC
I have attached the boot log "Created attachment 157511 [details] boot OK with v2 patches"

Boot OK, but sometimes it print:
"UDP: bad checksum. From 128.224.179.59:1534 to 128.224.179.255:1534 ulen 261"

ixgbe and xhci can work.
Comment 13 Yinghai Lu 2014-11-20 00:48:22 UTC
Please use patch 1, 2, 3 on top of

git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git disable_usb_legacy_early_2014_11_19


https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=disable_usb_legacy_early_2014_11_19
Comment 14 Pengyu Ma 2014-11-20 03:49:32 UTC
Created attachment 158241 [details]
disable_usb_legacy_early_2014_11_19 with 2 patches boot ok

ixgbe and xhci work well.
Comment 15 Yinghai Lu 2014-11-20 03:56:54 UTC
Do you mean 
disable_usb_legacy_early_2014_11_19 +
patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch2: v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode

or
disable_usb_legacy_early_2014_11_19 +
patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch3: patch that force reassign that could cause xhci hang 

?
Comment 16 Pengyu Ma 2014-11-20 05:08:28 UTC
With:
patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch2: v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode

No patch3, I will test patch 3 now.
Comment 17 Pengyu Ma 2014-11-20 05:33:18 UTC
Created attachment 158251 [details]
disable_usb_legacy_early_2014_11_19 with 3 patches boot hang

with patches on branch disable_usb_legacy_early_2014_11_19:

patch1: PCI: Don't reject 64bit mmio on 32bit/PAE mode
patch2: v2 x86, PCI: support mmio more than 44 bit on 32bit/PAE mode
patch3: patch that force reassign that could cause xhci hang 

Boot hang after:
calling  quirk_usb_early_handoff+0x0/0x1f0 @ 1 for 0000:00:14.0
Comment 18 Yinghai Lu 2014-11-20 07:13:08 UTC
ok, then we will have to use patch1+patch2.

patch 1 will be mainline via pci/for-linus.
patch 2 may need some time...
Comment 19 Bjorn Helgaas 2015-03-10 17:45:35 UTC
I think this should be resolved by:

  7fc986d8a972 PCI: Support 64-bit bridge windows if we have 64-bit dma_addr_t

which appeared in v3.18.  I'm closing this on that assumption.  If this is still a problem on v3.18 or later, please reopen or report as a new bug.