The above looks exactly like the sort of trace I'd expect if the server
is failing to reply to my RPC request.

If this is RPCSEC_GSS, are you perhaps enabling 3des or aes encryption against
a server that doesn't support it?

(In reply to comment #1)
> The above looks exactly like the sort of trace I'd expect if the server
> is failing to reply to my RPC request.

Certainly sounds reasonable.
 
> If this is RPCSEC_GSS, are you perhaps enabling 3des or aes encryption
> against
> a server that doesn't support it?

The *only* thing I am doing is changing which kernel I boot.  How would I derive from the different kernels if this type of scenario is what is happening?

Hmm... I was hoping that rpc.gssd would log what enctype it uses, but it doesn't appear to do so.

One thing you might try is simply to comment out the call to
'parse_enctypes()' in utils/gssd/gssd_proc.c:handle_gssd_upcall() and then
recompile your nfs-utils.

Since I hate to assume, let me confirm that you want that nfs-utils change on the server right?

(In reply to comment #3)
> 
> One thing you might try is simply to comment out the call to
> 'parse_enctypes()' in utils/gssd/gssd_proc.c:handle_gssd_upcall() and then
> recompile your nfs-utils.

Is that on the client or the server?  I'm thinking client since it's in utils/gssd/.

What are the results I am looking for when I've done that (and restarted gssd on the client)?

(In reply to comment #3)
> 
> One thing you might try is simply to comment out the call to
> 'parse_enctypes()' in utils/gssd/gssd_proc.c:handle_gssd_upcall() and then
> recompile your nfs-utils.

Hrm.  I don't have a parse_enctypes() in my utils/gssd/gssd_proc.c:handle_gssd_upcall().  I am using nfs-utils-1.2.2.

Ah... What version of kerberos?

Also, what does 'sudo klist -keK' show on the client and server?

(In reply to comment #6)
> Ah... What version of kerberos?

1.8.1 on both the client and server.  Specifically 1.8.1+dfsg-5ubuntu0.6 and 1.8.1+dfsg-2ubuntu0.6 respectively, for whatever it's worth.
 
> Also, what does 'sudo klist -keK' show on the client and server?

Client:
$ sudo klist -keK
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/pc.example.com@ILINX (Triple DES cbc mode with HMAC/sha1)  (0x3438...f462)
   2 nfs/pc.example.com@ILINX (DES cbc mode with CRC-32)  (0x7c...d5)
   4 host/pc.example.com@ILINX (Triple DES cbc mode with HMAC/sha1)  (0x926d...52b3)
   4 host/pc.example.com@ILINX (DES cbc mode with CRC-32)  (0xdf...c4)

Server:
# klist -keK
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x98...51)
   5 imap/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x25...52)
   5 imap/linux.example.com@ILINX (Triple DES cbc mode with HMAC/sha1)  (0xfd0d...32b3)
   3 smtp/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x16...76)
   3 smtp/linux.example.com@ILINX (Triple DES cbc mode with HMAC/sha1)  (0xc8b6...4952)
   3 host/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0xd0...e5)
   3 host/linux.example.com@ILINX (Triple DES cbc mode with HMAC/sha1)  (0x6745...464a)

Can you try getting rid of the Triple DES 'nfs' and 'host' service entries on the
client (and possibly on the server too) keytab?

Then please run the rpc.gssd with the '-f -vvv' options on the client side to ensure that you get a full log of what happens when you try to mount.

OK.  Removed the 3des entries:

Client:
pc# klist -keK
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/pc.example.com@ILINX (DES cbc mode with CRC-32)  (0x7c...d5)
   4 host/pc.example.com@ILINX (DES cbc mode with CRC-32)  (0xdf...c4)

Server:
linux# klist -keK
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x98...51)
   5 imap/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x25...52)
   3 smtp/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0x16...76)
   3 host/linux.example.com@ILINX (DES cbc mode with CRC-32)  (0xd0...e5)

pc# rpc.gssd with the -f -vvv
beginning poll
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt6e1)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt6e1)
process_krb5_upcall: service is '<null>'
Full hostname for 'linux.example.com' is 'linux.example.com'
Full hostname for 'pc' is 'pc'
Key table entry not found while getting keytab entry for 'root/pc@ILINX'
Key table entry not found while getting keytab entry for 'nfs/pc@ILINX'
Key table entry not found while getting keytab entry for 'host/pc@ILINX'
Success getting keytab entry for nfs/*@ILINX
WARNING: Key table entry not found while getting initial ticket for principal 'nfs/pc.example.com@ILINX' using keytab 'WRFILE:/etc/krb5.keytab'
ERROR: No credentials found for connection to server linux.example.com
doing error downcall
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6e1
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6e0
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6df
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6e4
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6e3

pc$ sudo mount -t nfs4 -o sec=krb5 linux:/tmp /mnt/tmp
mount.nfs4: access denied by server while mounting linux:/tmp

I wasn't getting access denied before removing the 3des keytab entries.

I also tries setting the hostname to the FQDN on the client given the number of errors about not being able to find keytab entries for "pc" and the end result was the same:

pc# rpc.gssd with the -f -vvv
beginning poll
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt75a)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt75a)
process_krb5_upcall: service is '<null>'
Full hostname for 'linux.example.com' is 'linux.example.com'
Full hostname for 'pc.example.com' is 'pc.example.com'
Key table entry not found while getting keytab entry for 'root/pc.example.com@ILINX'
Success getting keytab entry for 'nfs/pc.example.com@ILINX'
WARNING: Key table entry not found while getting initial ticket for principal 'nfs/pc.example.com@ILINX' using keytab 'WRFILE:/etc/krb5.keytab'
ERROR: No credentials found for connection to server linux.example.com
doing error downcall
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt75a
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt754
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt753
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt75d
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt75c

for

pc$ sudo mount -t nfs4 -o sec=krb5 linux:/tmp /mnt/tmp
mount.nfs4: access denied by server while mounting linux:/tmp

Although, as I said before, this now getting "access denied" is new since having removed the 3des entries and all of this, (client named "pc", 3des entries in keytabs, etc.) all work on older kernels.

Thanx much for all of the input so far.  Is there anything more I can provide to help move this along to resolution?

You probably saw this on the linux nfs list but just to keep information here complete (from http://permalink.gmane.org/gmane.linux.nfs/39578), when I posted the output of both the rpc.gssd and rpc.svcgssd services Kevin Coffman determined:

> prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and leng=
th 8

This shows that a des (enctype 4) session key is being negotiated and
delivered to both kernels, so I don't think any of the Kerberos issues
should be involved here.  (At least from the user-land perspective.)

I'm not sure what kernel change would be causing your hang ...