Bug 10893

Summary: alpha: Can't access PCI device resources through sysfs
Product: Drivers Reporter: Matt Turner (mattst88)
Component: PCIAssignee: Ivan Kokshaysky (ink)
Status: RESOLVED CODE_FIX    
Severity: normal CC: armin76, bunk, idr, ink, klausman, martisch, wlgivens
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.24 Subsystem:
Regression: No Bisected commit-id:
Attachments: first cut on pci sysfs resources

Description Matt Turner 2008-06-11 09:26:15 UTC
Latest working kernel version: none
Earliest failing kernel version: 
Distribution: Gentoo
Hardware Environment: Alphaserver DS20L, Radeon 9100 PCI
Software Environment: gentoo-sources-2.6.24-r3
Problem Description:

On alpha, access to PCI resources through sysfs is not implemented, or not working. /sys/devices/pci/<pciid>/resourceX does not exist.

Access to PCI resources through sysfs is necessary for the upcoming xserver.

See Documentation/filesystems/sysfs-pci.txt

Please implement.
Comment 1 Tobias Klausmann 2008-06-11 11:11:08 UTC
Note that very, very few alphas have AGP ports. None that I know of have PCI Express ports. As such, most of the alpha userbase has PCI cards - which don't work at all wihtout PCI resources.
Comment 2 Jesse Barnes 2008-06-11 16:05:30 UTC
We'll need to find one of the Linux alpha hackers to implement this.  The internal kernel interfaces are pretty simple, but alpha machines have a nontrivial I/O topology, so there may be some arch specific surgery required.
Comment 3 Jesse Barnes 2008-06-12 11:43:44 UTC
rth owns #10652, so he should probably own this one too.  I don't think the implementation will be too difficult, you can look at Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code for an example (arch/ia64/pci/pci.c:pci_mmap_page_range.
Comment 4 Matt Turner 2008-06-12 12:33:52 UTC
Since I filed bug 10652, I haven't received any indication that anyone has even seen that bug. :( The only reason he's the owner is that he's the default for alpha specific bugs.

I mailed lkml a few days before filing the bug and even CC'd rth and Ivan Kokshaysky but never received any kind of reply.

I believe bug 10652 is a symptom of this bug. I have to imagine that if this one is fixed, the other will be at the same time.

I'll mail axp-list to see if I can find some alpha hackers to take a look at this bug.
Comment 5 Jesse Barnes 2008-06-12 12:55:22 UTC
It might not be too hard to do yourself too, assuming you're ok with building kernels and crashing your box a few times. :)  There's already some code in arch/alpha/kernel/pci.c to iomap PCI space, and there are several examples of pci_mmap_page_range in the tree (I'm not sure, it might even be trivial to get it working on your platform by copying one of the existing implementations).  If you give it a try and have questions, let me know and I'll do my best to help.
Comment 6 Adrian Bunk 2008-06-12 16:24:17 UTC
(In reply to comment #3)
> rth owns #10652, so he should probably own this one too.
>

rth is the default owner of all Alpha bugs in the Bugzilla.

> I don't think the
> implementation will be too difficult, you can look at
> Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code for
> an example (arch/ia64/pci/pci.c:pci_mmap_page_range.

Ivan, can you look at this bug?
Comment 7 Adrian Bunk 2008-06-12 16:25:17 UTC
(In reply to comment #6)
> (In reply to comment #3)
> > rth owns #10652, so he should probably own this one too.
> >
> 
> rth is the default owner of all Alpha bugs in the Bugzilla.

But I haven't seen him doing Alpha kernel stuff for some time.
Comment 8 Ivan Kokshaysky 2008-06-16 07:43:13 UTC
(In reply to comment #6)
> (In reply to comment #3)
> > I don't think the
> > implementation will be too difficult, you can look at
> > Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code
> for
> > an example (arch/ia64/pci/pci.c:pci_mmap_page_range.
> 
> Ivan, can you look at this bug?

Unfortunately, I don't see any practical way to implement PCI IO mmap on older alphas that lack byte and word load/store instructions, because there is no 1:1 mapping from IO address space to CPU address space. Actually there is sort of 1:32 mapping (so-called sparse addressing), where each PCI bus address is represented by 32 CPU addresses and low-order address bits determine PCI byte enables.

Sure, pci_mmap_page_range would be more or less trivial on newer machines (starting from ev56/pyxis), but I guess that X server should have some fallback mechanism anyway if mmap doesn't work (like it probably has for ioports).
Comment 9 Matt Turner 2008-06-16 08:04:40 UTC
I'll make sure Ian Romanick (the libpciaccess maintainer) sees this. Thanks for taking a look at it.

If a fallback isn't acceptable for the X11 developers (it may not be), can PCI IO mmap be implemented for BWX-enabled Alphas? By 'trivial', I assume you mean a relatively small amount of work.
Comment 10 Jesse Barnes 2008-06-16 09:58:22 UTC
Oh yuck, I forgot about alpha's load/store constraints.  One way of dealing with it might be to page fault on every access to a PCI sysfs resource file and take appropriate action.  Alternately, you could implement a read/write interface to the resource files instead of mmap, and document that if userspace gets an ENODEV from the mmap call, it should use read/write instead (this is what we did for the legacy_io file since ia64 has similar limitations there).
Comment 11 Ivan Kokshaysky 2008-06-16 15:38:48 UTC
(In reply to comment #10)
> Oh yuck, I forgot about alpha's load/store constraints.  One way of dealing
> with it might be to page fault on every access to a PCI sysfs resource file
> and
> take appropriate action.  Alternately, you could implement a read/write
> interface to the resource files instead of mmap, and document that if
> userspace
> gets an ENODEV from the mmap call, it should use read/write instead (this is
> what we did for the legacy_io file since ia64 has similar limitations there).

No, both of these methods would be unacceptably slow for X. mmap is the only way to go (currently xserver does mmap on /dev/mem).
It would be good if we could create resourceN_sparse/dense files (instead of resourceN_wc files which are totally useless on alpha, as there is always write-combining unless you put explicit memory barriers between the writes).
But this requires alpha-specific version of the code under #ifdef HAVE_PCI_MMAP
in pci-sysfs.c...
Comment 12 Jesse Barnes 2008-06-16 16:30:51 UTC
Ugg, yeah doing page faulting would probably be too slow, and read/write would definitely be slower than direct access, but another resourceN file?  I guess that's fine for alpha, it seems a lot of userspace packages support the sparse method...

But where can I read up on the dense stuff?  Would the current resourceN files be suitable as long as they were only read/written with 32 bit load/stores with 32 bit alignment?

As for resourceN_wc, it's fine if it's not available on alpha or has the same behavior as the regular resourceN file.
Comment 13 Ivan Kokshaysky 2008-06-20 15:05:16 UTC
(In reply to comment #12)
> But where can I read up on the dense stuff?  Would the current resourceN
> files
> be suitable as long as they were only read/written with 32 bit load/stores
> with
> 32 bit alignment?

Yes (and aligned 64-bit accesses work as well). You can safely use it if all MMIO registers are 32-bit, or for memory-like regions like framebuffers.
However, I'd prefer to have _dense suffix for those, and use 'normal' resourceN files for bwx alphas only.

So I'd suggest a single resourceN file per resource for bwx alphas (1:1 mapping for both MEM and IO, no access constrains);
for older machines: one resourceN_sparse file per IO resource (there is no dense  mapping for IO), and either both _sparse and _dense files for MEM, or just one _dense file in some cases (e.g. for prefetchable MEM or resources > 128M).

If this sounds reasonable, I could try to implement it for 2.6.27.
Thoughts?
Comment 14 Jesse Barnes 2008-06-20 15:26:35 UTC
Yeah, that's ok with me, I was just trying to think of a way that would keep alpha from having to use a bunch of special userspace code (like it does now); the more we can keep common, the more things will "just work" on alpha.  But I guess if it's just the newer machines with the common interfaces, that's good enough.

Thanks,
Jesse
Comment 15 Ivan Kokshaysky 2008-06-20 15:48:49 UTC
Fine. It seems that even on old hardware with odd mappings userspace could be much cleaner than it's now... OK, will work on that.
Comment 16 Matt Turner 2008-07-18 15:43:12 UTC
Are there any patches I can test?
Comment 17 Tobias Klausmann 2008-12-11 12:12:20 UTC
If there's any way we (Gentoo Alpha team) can help with testing or the like, feel free to bug us about it.
Comment 18 Martin Schmidt 2009-01-10 16:07:18 UTC
Since newer Alphas (>= ev56) are more likely to be used as workstations with Xorg and this is the easier architecture to fix, i would suggest to focus on those first. Then Xorg-7.4 could be tested and ev4/5 versions can be fixed up and cleaned later.

Thanks,
Martin
Comment 19 Ivan Kokshaysky 2009-02-04 04:20:14 UTC
Created attachment 20106 [details]
first cut on pci sysfs resources
Comment 20 Ivan Kokshaysky 2009-02-04 04:21:04 UTC
Sorry for delay, guys.

I've finally gotten around to write a code for this.

At the moment, it's tested on ev6 (normal resourceN files with
byte/word access): libpciaccess seems to work correctly for
both I/O and memory; mmapable legacy_io (for VGA registers and so on)
works as well.

Sparse/dense stuff is there, but it's completely untested yet -
I'm going to fire up my sx164 and hack the kernel a bit so it
would use sparse/dense I/O instead of bwx. Hopefully this weekend...

Patch appended for review. There is no /proc mmap support, and actually
I don't want to implement it, as it won't properly handle sparse/dense/bwx
by design.
Comment 21 Matt Turner 2009-02-04 17:24:17 UTC
I'm running the patch now. The mmap-or-not stuff is over my head, but the patch seems to operate cleanly.
Comment 22 Raúl Porcel 2009-03-07 11:18:26 UTC
Ping, hows this going?
Comment 23 Tobias Klausmann 2009-03-20 13:10:22 UTC
I had completely forgotten about this patch, sorry 'bout that.

UP1500 with an AGP nvidia card and the most recent xorg-1.5 nv driver: works great. Testing with my XP (and Permedia/glint) will take a bit but I'll do it this weekend.
Comment 24 Tobias Klausmann 2009-03-21 11:52:06 UTC
So the XP1000 works in principle. I can get a Millenium II to work and all would be well - except the glint driver (Permedia) is broken in some way:

$ startx
xauth:  creating new authority file /home/klausman/.serverauth.1326


X.Org X Server 1.5.3
Release Date: 5 November 2008
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.27-gentoo-r5 alpha 
Current Operating System: Linux dragunov 2.6.29-rc8 #4 Sat Mar 21 18:22:14 CET 2009 alpha
Build Date: 21 March 2009  03:28:48PM
 
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat Mar 21 19:25:55 2009
(==) Using config file: "/etc/X11/xorg.conf"

Backtrace:
0: X(xorg_backtrace+0x4c) [0x12011067c]
1: X(xf86SigHandler+0x60) [0x1200a1110]
2: /lib/libc.so.6.1 [0x2000058af50]
3: X(readDense32+0x8) [0x12007f058]
4: /usr/lib/xorg/modules/drivers//glint_drv.so(Permedia2I2CPutBits+0x64) [0x2000088e6d4]
5: X [0x1200d80e8]
6: X [0x1200d81d4]
7: X [0x1200d87ec]
8: X [0x1200d8988]
9: X(xf86I2CWriteRead+0x1c) [0x1200d8aac]
10: X(xf86DoEDID_DDC2+0x180) [0x1200d4950]
11: /usr/lib/xorg/modules/drivers//glint_drv.so [0x2000088269c]
12: X(InitOutput+0xfb4) [0x120072c74]
13: X(main+0x35c) [0x12003055c]
14: /lib/libc.so.6.1(__libc_start_main+0xec) [0x2000057202c]
15: X(__start+0x38) [0x12002f918]

Fatal server error:
Caught signal 11.  Server aborting


^Cgiving up.
xinit:  No such file or directory (errno 2):  unable to connect to X server
xinit:  No such process (errno 3):  unexpected signal 2.

I doubt this is the fault of the patch. I'll try to find some Xorg dev to take a look at it.
Comment 25 Matt Turner 2009-10-29 20:31:35 UTC
This is fixed in commit 10a0ef39fbd1d484c2bbc1ffd83d57ecef209140 before 2.6.30. Closing as resolved.