Bug 117191 - BAR allocation problems on sparc
Summary: BAR allocation problems on sparc
Status: NEW
Alias: None
Product: Drivers
Classification: Unclassified
Component: PCI (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: drivers_pci@kernel-bugs.osdl.org
URL: http://lkml.kernel.org/r/alpine.LRH.2...
Keywords:
Depends on:
Blocks:
 
Reported: 2016-04-25 20:36 UTC by Bjorn Helgaas
Modified: 2018-07-12 11:03 UTC (History)
2 users (show)

See Also:
Kernel Version: 4.3
Subsystem:
Regression: Yes
Bisected commit-id:


Attachments
dmesg - sb100 (19.04 KB, text/plain)
2016-04-25 20:40 UTC, Bjorn Helgaas
Details
dmesg - v240 (21.65 KB, text/plain)
2016-04-25 20:40 UTC, Bjorn Helgaas
Details
dmesg - v210 (15.83 KB, text/plain)
2016-04-25 20:41 UTC, Bjorn Helgaas
Details
dmesg - v120 (17.43 KB, text/plain)
2016-04-25 20:41 UTC, Bjorn Helgaas
Details
dmesg - t1-200 (17.83 KB, text/plain)
2016-04-25 20:41 UTC, Bjorn Helgaas
Details
dmesg - t1-105 (17.81 KB, text/plain)
2016-04-25 20:42 UTC, Bjorn Helgaas
Details
lspci - v210 (23.64 KB, text/plain)
2016-04-25 20:42 UTC, Bjorn Helgaas
Details
dmesg - t2000 (20.78 KB, text/plain)
2016-10-28 10:41 UTC, Meelis Roos
Details
dmesg - Dell PE 1650 (39.19 KB, text/plain)
2016-11-01 07:57 UTC, Meelis Roos
Details
2018 dmesg of V100 (11.35 KB, text/plain)
2018-04-09 08:52 UTC, Meelis Roos
Details
2018 dmesg Netra X1 (12.11 KB, text/plain)
2018-04-09 08:53 UTC, Meelis Roos
Details
2018 lspci of Netra X1 (8.62 KB, text/plain)
2018-04-09 09:23 UTC, Meelis Roos
Details
2018 lspci of V100 (8.54 KB, text/plain)
2018-04-09 09:26 UTC, Meelis Roos
Details
2018 dmesg of T2000 (21.30 KB, text/plain)
2018-04-09 10:21 UTC, Meelis Roos
Details
2018 lspci of T2000 (69.12 KB, text/plain)
2018-04-09 10:22 UTC, Meelis Roos
Details
2018 dmesg of V245 (21.31 KB, text/plain)
2018-04-09 10:23 UTC, Meelis Roos
Details
2018 lspci of V245 (69.63 KB, text/plain)
2018-04-09 10:24 UTC, Meelis Roos
Details
2018 dmesg of V440 (16.53 KB, text/plain)
2018-04-09 10:25 UTC, Meelis Roos
Details
2018 lspci of V440 (13.37 KB, text/plain)
2018-04-09 10:25 UTC, Meelis Roos
Details
2018 dmesg of V240 (23.19 KB, text/plain)
2018-04-09 10:28 UTC, Meelis Roos
Details
2018 lspci of V240 (18.86 KB, text/plain)
2018-04-09 10:29 UTC, Meelis Roos
Details
2018 dmesg of Netra T1-105 (15.99 KB, text/plain)
2018-04-09 10:30 UTC, Meelis Roos
Details
2018 lspci of Netra T1-105 (17.09 KB, text/plain)
2018-04-09 10:30 UTC, Meelis Roos
Details
2018 dmesg of Netra T1-200 (15.42 KB, text/plain)
2018-04-09 10:32 UTC, Meelis Roos
Details
2018 lspci of Netra T1-200 (18.61 KB, text/plain)
2018-04-09 10:32 UTC, Meelis Roos
Details
2018 dmesg of V120 (15.07 KB, text/plain)
2018-04-09 10:33 UTC, Meelis Roos
Details
2018 lspci of V120 (17.02 KB, text/plain)
2018-04-09 10:34 UTC, Meelis Roos
Details
2018 dmesg of V210 (26.59 KB, text/plain)
2018-04-09 10:35 UTC, Meelis Roos
Details
2018 lspci of V210 (16.74 KB, text/plain)
2018-04-09 10:35 UTC, Meelis Roos
Details
2018 dmesg of Blade 100 (16.41 KB, text/plain)
2018-04-09 10:37 UTC, Meelis Roos
Details
2018 lspci of Blade 100 (18.63 KB, text/plain)
2018-04-09 10:38 UTC, Meelis Roos
Details
Reserve VGA framebuffer only if VGA device is present (5.43 KB, patch)
2018-05-21 18:46 UTC, Bjorn Helgaas
Details | Diff
Reserve VGA framebuffer only if VGA device is present (v2) (27.17 KB, patch)
2018-05-22 17:25 UTC, Bjorn Helgaas
Details | Diff
v215 lspci short (1.68 KB, text/plain)
2018-06-30 07:24 UTC, Anatoly Pugachev
Details
v215, dmesg output 4.18.0-rc2 (26.78 KB, text/plain)
2018-06-30 08:08 UTC, Anatoly Pugachev
Details
v215 lspci long, 4.18.0-rc2 (46.76 KB, text/plain)
2018-06-30 08:10 UTC, Anatoly Pugachev
Details
dmesg on T2000 with 4.17.0 (30.07 KB, text/plain)
2018-07-06 19:42 UTC, Meelis Roos
Details
dmesg on V245 with 4.17.0 (32.68 KB, text/plain)
2018-07-06 19:43 UTC, Meelis Roos
Details
dmesg on V245 with 4.18.0-rc2-00169-g90368a3-dirty (39.61 KB, text/plain)
2018-07-06 19:46 UTC, Meelis Roos
Details
dmesg on T2000 with 4.18.0-rc2-00132-gf574943-dirty (35.17 KB, text/plain)
2018-07-06 19:49 UTC, Meelis Roos
Details
dmesg on v215 with 4.18.0-rc3-sparc64-smp (debian kernel) (23.34 KB, text/plain)
2018-07-12 11:02 UTC, Anatoly Pugachev
Details

Description Bjorn Helgaas 2016-04-25 20:36:53 UTC
Meelis reports BAR allocation problems on various machines: http://lkml.kernel.org/r/alpine.LRH.2.20.1510071147520.8345@math.ut.ee

Attaching sparc dmesg logs here for archival purposes.
Comment 1 Bjorn Helgaas 2016-04-25 20:40:23 UTC
Created attachment 214231 [details]
dmesg - sb100
Comment 2 Bjorn Helgaas 2016-04-25 20:40:46 UTC
Created attachment 214241 [details]
dmesg - v240
Comment 3 Bjorn Helgaas 2016-04-25 20:41:08 UTC
Created attachment 214251 [details]
dmesg - v210
Comment 4 Bjorn Helgaas 2016-04-25 20:41:29 UTC
Created attachment 214261 [details]
dmesg - v120
Comment 5 Bjorn Helgaas 2016-04-25 20:41:52 UTC
Created attachment 214271 [details]
dmesg - t1-200
Comment 6 Bjorn Helgaas 2016-04-25 20:42:10 UTC
Created attachment 214281 [details]
dmesg - t1-105
Comment 7 Bjorn Helgaas 2016-04-25 20:42:28 UTC
Created attachment 214291 [details]
lspci - v210
Comment 8 Bjorn Helgaas 2016-10-27 16:22:08 UTC
These issues still exist in 4.8 per Meelis: http://lkml.kernel.org/r/alpine.LRH.2.20.1610271756330.32157@math.ut.ee

Dmesg snippets from that email:

> > sparc64 machines:
> >
> > http://kodu.ut.ee/~mroos/dm/dm.t1-105

Still present in 4.8.0-09031-g95107b3:

[   21.143842] pci 0000:00:01.0: can't claim BAR 8 [mem 0x1ff00000000-0x1ffbfffffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]
[   21.143947] pci 0000:02:01.0: can't claim BAR 8 [mem 0x1ff00100000-0x1ff028fffff]: no compatible bridge window
[   21.144020] pci 0000:03:0f.0: can't claim BAR 8 [mem 0x1ff00100000-0x1ff028fffff]: no compatible bridge window
[   21.144087] pci 0000:04:00.0: can't claim BAR 0 [mem 0x1ff01000000-0x1ff01ffffff]: no compatible bridge window
[   21.144153] pci 0000:04:00.0: can't claim BAR 1 [mem 0x1ff02000000-0x1ff027fffff]: no compatible bridge window
[   21.144220] pci 0000:04:00.1: can't claim BAR 0 [mem 0x1ff02800000-0x1ff0280702f]: no compatible bridge window
[   21.144288] pci 0000:04:04.0: can't claim BAR 1 [mem 0x1ff02808000-0x1ff02808fff]: no compatible bridge window
[   21.144355] pci 0000:04:04.0: can't claim BAR 6 [mem 0x1ff02810000-0x1ff0281ffff]: no compatible bridge window
...
[   50.775038] qla1280 0000:04:04.0: can't ioremap BAR 1: [mem size 0x00001000]
[   50.883435] qla1280: Unable to map I/O memory

> > http://kodu.ut.ee/~mroos/dm/dm.t1-200

Still present in 4.8.0-09031-g95107b3:

[   51.138280] pci 0000:00:01.0: can't claim BAR 8 [mem 0x1ff00000000-0x1ffbfffffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]
[   51.138399] pci 0000:02:08.0: can't claim BAR 1 [mem 0x1ff00002000-0x1ff00003fff 64bit]: no compatible bridge window
[   51.138461] pci 0000:02:08.0: can't claim BAR 3 [mem 0x1ff00004000-0x1ff00005fff 64bit]: no compatible bridge window
[   51.138522] pci 0000:02:08.1: can't claim BAR 1 [mem 0x1ff00006000-0x1ff00007fff 64bit]: no compatible bridge window
[   51.138580] pci 0000:02:08.1: can't claim BAR 3 [mem 0x1ff00008000-0x1ff00009fff 64bit]: no compatible bridge window
[   51.138639] pci 0000:02:05.0: can't claim BAR 1 [mem 0x1ff0000a000-0x1ff0000bfff]: no compatible bridge window
[   51.138695] pci 0000:02:05.0: can't claim BAR 6 [mem 0x1ff00020000-0x1ff0003ffff]: no compatible bridge window

> > http://kodu.ut.ee/~mroos/dm/dm.v120

Still present in 4.8.0-09031-g95107b3:

[   51.025122] pci 0000:00:01.0: can't claim BAR 8 [mem 0x1ff00000000-0x1ffbfffffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]
[   51.025240] pci 0000:02:08.0: can't claim BAR 1 [mem 0x1ff00002000-0x1ff00003fff 64bit]: no compatible bridge window
[   51.025302] pci 0000:02:08.0: can't claim BAR 3 [mem 0x1ff00004000-0x1ff00005fff 64bit]: no compatible bridge window
[   51.025363] pci 0000:02:08.1: can't claim BAR 1 [mem 0x1ff00006000-0x1ff00007fff 64bit]: no compatible bridge window
[   51.025423] pci 0000:02:08.1: can't claim BAR 3 [mem 0x1ff00008000-0x1ff00009fff 64bit]: no compatible bridge window

> > http://kodu.ut.ee/~mroos/dm/dm.v210

Still present in 4.8.0:

Oct  5 13:56:52 v210 kernel: [   39.003270] PCI: Claiming 0001:00:07.0: Resource 0: 000007fe01000000..000007fe0100ffff [101]
Oct  5 13:56:52 v210 kernel: [   39.003309] pci 0001:00:07.0: can't claim BAR 0 [io  0x7fe01000000-0x7fe0100ffff]: address conflict with 0001:00:06.0 [io  0x7fe01000600-0x7fe0100061f]Oct  5 13:56:52 v210 kernel: [   39.003350] PCI: Claiming 0001:00:07.0: Resource 1: 000007ff00000000..000007ff000fffff [200]
Oct  5 13:56:52 v210 kernel: [   39.003389] pci 0001:00:07.0: can't claim BAR 1 [mem 0x7ff00000000-0x7ff000fffff]: address conflict with Video RAM area [??? 0x7ff000a0000-0x7ff000bffff flags 0x80000000]
Oct  5 13:56:52 v210 kernel: [   39.003437] PCI: Claiming 0001:00:07.0: Resource 2: 000007ff00000000..000007ff000fffff [200]
Oct  5 13:56:52 v210 kernel: [   39.003474] pci 0001:00:07.0: can't claim BAR 2 [mem 0x7ff00000000-0x7ff000fffff]: address conflict with Video RAM area [??? 0x7ff000a0000-0x7ff000bffff flags 0x80000000]


> > http://kodu.ut.ee/~mroos/dm/dm.v240

Still present in 4.8.0-09031-g95107b3:

[   41.120991] pci 0001:00:07.0: can't claim BAR 0 [io  0x7fe01000000-0x7fe0100ffff]: address conflict with 0001:00:06.0 [io  0x7fe01000600-0x7fe0100061f]
[   41.121054] pci 0001:00:07.0: can't claim BAR 1 [mem 0x7ff00000000-0x7ff000fffff]: address conflict with Video RAM area [??? 0x7ff000a0000-0x7ff000bffff flags 0x80000000]
[   41.121122] pci 0001:00:07.0: can't claim BAR 2 [mem 0x7ff00000000-0x7ff000fffff]: address conflict with Video RAM area [??? 0x7ff000a0000-0x7ff000bffff flags 0x80000000]

> > http://kodu.ut.ee/~mroos/dm/dm.sb100

Still present in 4.9.0-rc1-00067-g1a1891d:

[   41.280647] pci 0000:00:07.0: can't claim BAR 0 [io  0x1fe02000000-0x1fe0200ffff]: address conflict with 0000:00:03.0 [io  0x1fe02000600-0x1fe0200061f]
[   41.282521] pci 0000:00:07.0: can't claim BAR 1 [mem 0x1ff00000000-0x1ff000fffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]

> > amd64 machine:
> >
> > http://kodu.ut.ee/~mroos/dm/dm.x2100

Still present in 4.8.0:

[    0.242410] pci 0000:00:01.1: can't claim BAR 4 [io  0x1c00-0x1c3f]: no compatible bridge window
[    0.242609] pci 0000:00:01.1: can't claim BAR 5 [io  0x1c40-0x1c7f]: no compatible bridge window
Comment 9 Meelis Roos 2016-10-28 10:41:29 UTC
Created attachment 243001 [details]
dmesg - t2000
Comment 10 Meelis Roos 2016-10-28 10:42:27 UTC
Found similar issue on Sparc T2000 with 4.9-rc2+git. Created an attachment for that too.
Comment 11 Meelis Roos 2016-10-28 10:44:08 UTC
My logs show that the t2000 issue was there at least in 4.7.0-07753-gc9b95e5. Earlier logs are not available.
Comment 12 Meelis Roos 2016-11-01 07:57:28 UTC
Created attachment 243431 [details]
dmesg - Dell PE 1650

Another x86 that is showing BAR allocation failure in 4.9.0-rc2-00138-g14970f2 (Dell PowerEdge 1650).
Comment 13 Meelis Roos 2018-04-09 08:52:21 UTC
Created attachment 275201 [details]
2018 dmesg of V100
Comment 14 Meelis Roos 2018-04-09 08:53:44 UTC
Created attachment 275203 [details]
2018 dmesg Netra X1
Comment 15 Meelis Roos 2018-04-09 09:23:26 UTC
Created attachment 275205 [details]
2018 lspci of Netra X1
Comment 16 Meelis Roos 2018-04-09 09:26:53 UTC
Created attachment 275207 [details]
2018 lspci of V100
Comment 17 Meelis Roos 2018-04-09 10:21:25 UTC
Created attachment 275209 [details]
2018 dmesg of T2000
Comment 18 Meelis Roos 2018-04-09 10:22:16 UTC
Created attachment 275211 [details]
2018 lspci of T2000
Comment 19 Meelis Roos 2018-04-09 10:23:52 UTC
Created attachment 275213 [details]
2018 dmesg of V245
Comment 20 Meelis Roos 2018-04-09 10:24:15 UTC
Created attachment 275215 [details]
2018 lspci of V245
Comment 21 Meelis Roos 2018-04-09 10:25:36 UTC
Created attachment 275217 [details]
2018 dmesg of V440
Comment 22 Meelis Roos 2018-04-09 10:25:58 UTC
Created attachment 275219 [details]
2018 lspci of V440
Comment 23 Meelis Roos 2018-04-09 10:28:27 UTC
Created attachment 275221 [details]
2018 dmesg of V240
Comment 24 Meelis Roos 2018-04-09 10:29:21 UTC
Created attachment 275223 [details]
2018 lspci of V240
Comment 25 Meelis Roos 2018-04-09 10:30:32 UTC
Created attachment 275225 [details]
2018 dmesg of Netra T1-105
Comment 26 Meelis Roos 2018-04-09 10:30:55 UTC
Created attachment 275227 [details]
2018 lspci of Netra T1-105
Comment 27 Meelis Roos 2018-04-09 10:32:19 UTC
Created attachment 275229 [details]
2018 dmesg of Netra T1-200
Comment 28 Meelis Roos 2018-04-09 10:32:40 UTC
Created attachment 275231 [details]
2018 lspci of Netra T1-200
Comment 29 Meelis Roos 2018-04-09 10:33:34 UTC
Created attachment 275233 [details]
2018 dmesg of V120
Comment 30 Meelis Roos 2018-04-09 10:34:05 UTC
Created attachment 275235 [details]
2018 lspci of V120
Comment 31 Meelis Roos 2018-04-09 10:35:01 UTC
Created attachment 275237 [details]
2018 dmesg of V210
Comment 32 Meelis Roos 2018-04-09 10:35:23 UTC
Created attachment 275239 [details]
2018 lspci of V210
Comment 33 Meelis Roos 2018-04-09 10:37:46 UTC
Created attachment 275241 [details]
2018 dmesg of Blade 100
Comment 34 Meelis Roos 2018-04-09 10:38:12 UTC
Created attachment 275243 [details]
2018 lspci of Blade 100
Comment 35 Bjorn Helgaas 2018-04-20 19:36:03 UTC
(In reply to Meelis Roos from comment #13)
> Created attachment 275201 [details]
> 2018 dmesg of V100

This shows a conflict with the legacy VGA framebuffer at PCI bus addresses 0xa0000-0xbffff:

  pci_bus 0000:00: root bus resource [mem 0x1ff00000000-0x1ffffffffff] (bus address [0x00000000-0xffffffff])
  pci 0000:00:05.0: can't claim BAR 6 [mem 0x1ff00080000-0x1ff000bffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]

This happens on all the machines here except T2000 and V245.  It happens because the sparc arch code unconditionally requests the framebuffer area before we can for PCI devices, and some of those devices might have BARs that overlap the framebuffer area.

In this V100 case, attachment 275207 [details] shows that the 00:05.0 expansion ROM is assigned space that overlaps the framebuffer:

  00:05.0 21x4x DEC-Tulip compatible 10/100 Ethernet (rev 31)
	Expansion ROM at 1ff00080000 [disabled] [size=256K]

The code paths involved are:

  pci_fire_pbm_init, schizo_pbm_init, pci_sun4v_pbm_init, psycho_pbm_init_common
    pci_determine_mem_io_space
      pci_register_legacy_regions
        p->start = mem_res->start + 0xa0000
        request_resource(mem_res, p)    # claim VGA framebuffer
    pci_scan_one_pbm
      pci_of_scan_bus                   # scan DT for PCI devices
      pci_claim_bus_resources           # claim PCI device BARs
Comment 36 Bjorn Helgaas 2018-05-21 18:46:03 UTC
Created attachment 276101 [details]
Reserve VGA framebuffer only if VGA device is present

This should resolve warnings like this:

pci 0002:00:07.0: can't claim BAR 1 [mem 0x7ff00000000-0x7ff000fffff]: address conflict with Video RAM area [??? 0x7ff000a0000-0x7ff000bffff flags 0x80000000]

which occurred on these systems: V100, Netra X1, V440, V240, Netra T1-105, Netra T1-200, V120, V210, Blade 100
Comment 37 Anatoly Pugachev 2018-05-22 13:59:42 UTC
JFYI,

sun v215 hardware: 

$ dmesg | grep BAR
[    5.077808] pci 0000:05:1e.0: can't claim BAR 0 [io  0x7f810000000-0x7f810000fff]: no compatible bridge window

# uname -a
Linux v215 4.16.0-1-sparc64-smp #1 SMP Debian 4.16.5-1 (2018-04-29) sparc64 GNU/Linux
Comment 38 Bjorn Helgaas 2018-05-22 17:25:24 UTC
Created attachment 276137 [details]
Reserve VGA framebuffer only if VGA device is present (v2)

(In reply to Anatoly Pugachev from comment #37)
> sun v215 hardware: 
> 
> $ dmesg | grep BAR
> [    5.077808] pci 0000:05:1e.0: can't claim BAR 0 [io 
> 0x7f810000000-0x7f810000fff]: no compatible bridge window
> 
> # uname -a
> Linux v215 4.16.0-1-sparc64-smp #1 SMP Debian 4.16.5-1 (2018-04-29) sparc64
> GNU/Linux

4.16 does not contain the proposed fix, so I don't expect any improvement there.

If it's convenient, you could test the attached patch (based on v4.17-rc1) and attach the complete dmesg log and "sudo lspci -vv" output.

I think it's likely the warning you're seeing will still be there as a result of a conflict with an M7101 device for which a quirk claims some space.

Is there something that doesn't work, or is the message itself the only problem?
Comment 39 Anatoly Pugachev 2018-06-30 07:24:04 UTC
Bjorn, 

there's nothing in pci-e slots on this machine. But per your last email on sparclinux@ i'm attaching dmesg and lspci output from v215 on kernel 4.18.0-rc2-00169-g90368a37fbbe
Comment 40 Anatoly Pugachev 2018-06-30 07:24:59 UTC
Created attachment 277071 [details]
v215 lspci  short
Comment 41 Anatoly Pugachev 2018-06-30 08:08:22 UTC
Created attachment 277073 [details]
v215, dmesg output 4.18.0-rc2
Comment 42 Anatoly Pugachev 2018-06-30 08:10:13 UTC
Created attachment 277075 [details]
v215 lspci long, 4.18.0-rc2
Comment 43 Meelis Roos 2018-07-06 19:42:55 UTC
Created attachment 277243 [details]
dmesg on T2000 with 4.17.0
Comment 44 Meelis Roos 2018-07-06 19:43:25 UTC
Created attachment 277245 [details]
dmesg on V245 with 4.17.0
Comment 45 Meelis Roos 2018-07-06 19:46:27 UTC
Created attachment 277247 [details]
dmesg on V245 with 4.18.0-rc2-00169-g90368a3-dirty

-dirty means bpfilter compilation patch, not related to PCI I hope
Comment 46 Meelis Roos 2018-07-06 19:49:10 UTC
Created attachment 277249 [details]
dmesg on T2000 with 4.18.0-rc2-00132-gf574943-dirty

-dirty means bpfilter compilation patch, not related to PCI I hope
Comment 47 Anatoly Pugachev 2018-07-12 11:02:20 UTC
Created attachment 277339 [details]
dmesg on v215 with 4.18.0-rc3-sparc64-smp (debian kernel)
Comment 48 Anatoly Pugachev 2018-07-12 11:03:57 UTC
JFYI

booted to latest debian packaged kernel (experimental release), there's no BAR messages in the dmesg on my v215.

Note You need to log in before you can comment on or make changes to this bug.