If "port=" is not specified during an NFS/RDMA mount attempt, the mount command fails. Kernel log for "mount -o rdma,vers=4 server:/share /mnt": Jan 8 15:58:28 manet kernel: rpcrdma: connection to 10.0.0.1:2049 closed (-111) Jan 8 15:58:28 manet kernel: rpcrdma: connection to 10.0.0.1:2049 closed (-111) Jan 8 15:58:28 manet kernel: rpcrdma: connection to 10.0.0.1:2049 closed (-111) Jan 8 15:58:28 manet kernel: rpcrdma: connection to 10.0.0.1:2049 closed (-111) Kernel log for "mount -o rdma,vers=3 server:/share /mnt": Jan 8 15:59:13 manet kernel: rpcrdma: connection to 10.0.0.1:0 closed (-111) Jan 8 15:59:13 manet kernel: rpcrdma: connection to 10.0.0.1:0 closed (-111) Jan 8 15:59:13 manet kernel: rpcrdma: connection to 10.0.0.1:0 closed (-111) Jan 8 15:59:13 manet kernel: rpcrdma: connection to 10.0.0.1:0 closed (-111) When "proto=rdma" is specified, the default port should be 20049 (see RFC 5666). Port assignment recorded by IANA: http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=nfsrdma The kernel should set the port to 20049 if "proto=rdma" is specified and the user did not specify "port=".
Could we perhaps fix this in userland instead of hard-coding new server ports in the kernel? Basically, if the mount fails with ECONNREFUSED, then have the mount utility try to look up the 'nfsrdma' service in the /etc/services file, and pass that info to the mount syscall.
One more question: Is there a way for the server to advertise the RDMA service port using rpcbind?
(In reply to Trond Myklebust from comment #1) > Could we perhaps fix this in userland instead of hard-coding new server > ports in the kernel? > > Basically, if the mount fails with ECONNREFUSED, then have the mount utility > try to look up the 'nfsrdma' service in the /etc/services file, and pass > that info to the mount syscall. I don't see the need to retry...? If "proto=rdma" is specified now, the port setting is undefined for vers=3, and is wrong for vers=4. Solaris clients negotiate 1. rdma, 2. tcp, 3. udp (if vers<4). Should we assume in the long run Linux is going to try 1. tcp, 2. udp (if vers<4), 3. rdma ? Port 2049 is defined in RFCs for NFSv4 over TCP, and likewise, port 20049 is similarly defined in RFC 5666. If "port=" isn't specified by the admin, it is simple for mount.nfs to add the standard setting and be done with it. But that's not the way kernel handles this for NFSv4 over TCP, AFAICT.
(In reply to Trond Myklebust from comment #2) > One more question: Is there a way for the server to advertise the RDMA > service port using rpcbind? rpcinfo doesn't show an "rdma" registration for any RPC service on my Solaris server. So it wouldn't help when mounting existing Solaris servers. I can ask Talpey if he thinks getting rpcbind involved is wise. My impression is that the whole concept of port is meaningless for RDMA, and it should be treated the same way NFSv4 over TCP treats it: just use port 20049 unless otherwise specified.
All that said, I have a slight preference for handling this in the kernel, since it's part of an Internet standard (and thus not a policy) and this is how it already works for NFSv4 over TCP. But I'm not religious about it.
Created attachment 123061 [details] Potential kernel patch to address bz 68401 To keep the conversation moving, here's a patch that implements the new behavior in the kernel. We can also look at mount.nfs changes, if you like.
RFC5667 doesn't require that the service be registered with rpcbind, and for v4, specifically mandates that port without exception (or requirement for portmap to show the service).
This issue appears to have been addressed by commit 0dfbb5f05e78 ("NFS: Make "port=" mount option optional for RDMA mounts"), available in the v5.0 kernel.