Bug 10893

Note that very, very few alphas have AGP ports. None that I know of have PCI Express ports. As such, most of the alpha userbase has PCI cards - which don't work at all wihtout PCI resources.

We'll need to find one of the Linux alpha hackers to implement this.  The internal kernel interfaces are pretty simple, but alpha machines have a nontrivial I/O topology, so there may be some arch specific surgery required.

rth owns #10652, so he should probably own this one too.  I don't think the implementation will be too difficult, you can look at Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code for an example (arch/ia64/pci/pci.c:pci_mmap_page_range.

Since I filed bug 10652, I haven't received any indication that anyone has even seen that bug. :( The only reason he's the owner is that he's the default for alpha specific bugs.

I mailed lkml a few days before filing the bug and even CC'd rth and Ivan Kokshaysky but never received any kind of reply.

I believe bug 10652 is a symptom of this bug. I have to imagine that if this one is fixed, the other will be at the same time.

I'll mail axp-list to see if I can find some alpha hackers to take a look at this bug.

It might not be too hard to do yourself too, assuming you're ok with building kernels and crashing your box a few times. :)  There's already some code in arch/alpha/kernel/pci.c to iomap PCI space, and there are several examples of pci_mmap_page_range in the tree (I'm not sure, it might even be trivial to get it working on your platform by copying one of the existing implementations).  If you give it a try and have questions, let me know and I'll do my best to help.

(In reply to comment #3)
> rth owns #10652, so he should probably own this one too.
>

rth is the default owner of all Alpha bugs in the Bugzilla.

> I don't think the
> implementation will be too difficult, you can look at
> Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code for
> an example (arch/ia64/pci/pci.c:pci_mmap_page_range.

Ivan, can you look at this bug?

(In reply to comment #6)
> (In reply to comment #3)
> > rth owns #10652, so he should probably own this one too.
> >
> 
> rth is the default owner of all Alpha bugs in the Bugzilla.

But I haven't seen him doing Alpha kernel stuff for some time.

(In reply to comment #6)
> (In reply to comment #3)
> > I don't think the
> > implementation will be too difficult, you can look at
> > Documentation/filesystems/sysfs-pci.txt for some info, and the ia64 code
> for
> > an example (arch/ia64/pci/pci.c:pci_mmap_page_range.
> 
> Ivan, can you look at this bug?

Unfortunately, I don't see any practical way to implement PCI IO mmap on older alphas that lack byte and word load/store instructions, because there is no 1:1 mapping from IO address space to CPU address space. Actually there is sort of 1:32 mapping (so-called sparse addressing), where each PCI bus address is represented by 32 CPU addresses and low-order address bits determine PCI byte enables.

Sure, pci_mmap_page_range would be more or less trivial on newer machines (starting from ev56/pyxis), but I guess that X server should have some fallback mechanism anyway if mmap doesn't work (like it probably has for ioports).

I'll make sure Ian Romanick (the libpciaccess maintainer) sees this. Thanks for taking a look at it.

If a fallback isn't acceptable for the X11 developers (it may not be), can PCI IO mmap be implemented for BWX-enabled Alphas? By 'trivial', I assume you mean a relatively small amount of work.

Oh yuck, I forgot about alpha's load/store constraints.  One way of dealing with it might be to page fault on every access to a PCI sysfs resource file and take appropriate action.  Alternately, you could implement a read/write interface to the resource files instead of mmap, and document that if userspace gets an ENODEV from the mmap call, it should use read/write instead (this is what we did for the legacy_io file since ia64 has similar limitations there).

(In reply to comment #10)
> Oh yuck, I forgot about alpha's load/store constraints.  One way of dealing
> with it might be to page fault on every access to a PCI sysfs resource file
> and
> take appropriate action.  Alternately, you could implement a read/write
> interface to the resource files instead of mmap, and document that if
> userspace
> gets an ENODEV from the mmap call, it should use read/write instead (this is
> what we did for the legacy_io file since ia64 has similar limitations there).

No, both of these methods would be unacceptably slow for X. mmap is the only way to go (currently xserver does mmap on /dev/mem).
It would be good if we could create resourceN_sparse/dense files (instead of resourceN_wc files which are totally useless on alpha, as there is always write-combining unless you put explicit memory barriers between the writes).
But this requires alpha-specific version of the code under #ifdef HAVE_PCI_MMAP
in pci-sysfs.c...

Ugg, yeah doing page faulting would probably be too slow, and read/write would definitely be slower than direct access, but another resourceN file?  I guess that's fine for alpha, it seems a lot of userspace packages support the sparse method...

But where can I read up on the dense stuff?  Would the current resourceN files be suitable as long as they were only read/written with 32 bit load/stores with 32 bit alignment?

As for resourceN_wc, it's fine if it's not available on alpha or has the same behavior as the regular resourceN file.

(In reply to comment #12)
> But where can I read up on the dense stuff?  Would the current resourceN
> files
> be suitable as long as they were only read/written with 32 bit load/stores
> with
> 32 bit alignment?

Yes (and aligned 64-bit accesses work as well). You can safely use it if all MMIO registers are 32-bit, or for memory-like regions like framebuffers.
However, I'd prefer to have _dense suffix for those, and use 'normal' resourceN files for bwx alphas only.

So I'd suggest a single resourceN file per resource for bwx alphas (1:1 mapping for both MEM and IO, no access constrains);
for older machines: one resourceN_sparse file per IO resource (there is no dense  mapping for IO), and either both _sparse and _dense files for MEM, or just one _dense file in some cases (e.g. for prefetchable MEM or resources > 128M).

If this sounds reasonable, I could try to implement it for 2.6.27.
Thoughts?

Yeah, that's ok with me, I was just trying to think of a way that would keep alpha from having to use a bunch of special userspace code (like it does now); the more we can keep common, the more things will "just work" on alpha.  But I guess if it's just the newer machines with the common interfaces, that's good enough.

Thanks,
Jesse

Fine. It seems that even on old hardware with odd mappings userspace could be much cleaner than it's now... OK, will work on that.

Are there any patches I can test?

If there's any way we (Gentoo Alpha team) can help with testing or the like, feel free to bug us about it.

Since newer Alphas (>= ev56) are more likely to be used as workstations with Xorg and this is the easier architecture to fix, i would suggest to focus on those first. Then Xorg-7.4 could be tested and ev4/5 versions can be fixed up and cleaned later.

Thanks,
Martin

Created attachment 20106 [details]
first cut on pci sysfs resources

Sorry for delay, guys.

I've finally gotten around to write a code for this.

At the moment, it's tested on ev6 (normal resourceN files with
byte/word access): libpciaccess seems to work correctly for
both I/O and memory; mmapable legacy_io (for VGA registers and so on)
works as well.

Sparse/dense stuff is there, but it's completely untested yet -
I'm going to fire up my sx164 and hack the kernel a bit so it
would use sparse/dense I/O instead of bwx. Hopefully this weekend...

Patch appended for review. There is no /proc mmap support, and actually
I don't want to implement it, as it won't properly handle sparse/dense/bwx
by design.

I'm running the patch now. The mmap-or-not stuff is over my head, but the patch seems to operate cleanly.

Ping, hows this going?

I had completely forgotten about this patch, sorry 'bout that.

UP1500 with an AGP nvidia card and the most recent xorg-1.5 nv driver: works great. Testing with my XP (and Permedia/glint) will take a bit but I'll do it this weekend.

So the XP1000 works in principle. I can get a Millenium II to work and all would be well - except the glint driver (Permedia) is broken in some way:

$ startx
xauth:  creating new authority file /home/klausman/.serverauth.1326


X.Org X Server 1.5.3
Release Date: 5 November 2008
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.27-gentoo-r5 alpha 
Current Operating System: Linux dragunov 2.6.29-rc8 #4 Sat Mar 21 18:22:14 CET 2009 alpha
Build Date: 21 March 2009  03:28:48PM
 
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat Mar 21 19:25:55 2009
(==) Using config file: "/etc/X11/xorg.conf"

Backtrace:
0: X(xorg_backtrace+0x4c) [0x12011067c]
1: X(xf86SigHandler+0x60) [0x1200a1110]
2: /lib/libc.so.6.1 [0x2000058af50]
3: X(readDense32+0x8) [0x12007f058]
4: /usr/lib/xorg/modules/drivers//glint_drv.so(Permedia2I2CPutBits+0x64) [0x2000088e6d4]
5: X [0x1200d80e8]
6: X [0x1200d81d4]
7: X [0x1200d87ec]
8: X [0x1200d8988]
9: X(xf86I2CWriteRead+0x1c) [0x1200d8aac]
10: X(xf86DoEDID_DDC2+0x180) [0x1200d4950]
11: /usr/lib/xorg/modules/drivers//glint_drv.so [0x2000088269c]
12: X(InitOutput+0xfb4) [0x120072c74]
13: X(main+0x35c) [0x12003055c]
14: /lib/libc.so.6.1(__libc_start_main+0xec) [0x2000057202c]
15: X(__start+0x38) [0x12002f918]

Fatal server error:
Caught signal 11.  Server aborting


^Cgiving up.
xinit:  No such file or directory (errno 2):  unable to connect to X server
xinit:  No such process (errno 3):  unexpected signal 2.

I doubt this is the fault of the patch. I'll try to find some Xorg dev to take a look at it.

This is fixed in commit 10a0ef39fbd1d484c2bbc1ffd83d57ecef209140 before 2.6.30. Closing as resolved.