Q:

NFS issue: clients can mount shares as NFSv3 but not as NFSv4 -- or how to debug NFS?

Problem description
I have a file server running Debian. On it I have a few NFS shares. 
When I mount the shares from a client using NFSv3 (mount.nfs 10.0.0.51:/exports/video /mnt -o vers=3,soft,intr,timeo=10), it works. However, I would like to use NFSv4 because of improved security and performance.
When I try to mount an NFSv4 share on malbec the mount command just hangs and finally times out after 2 minutes.
How do I make the clients mount the NFSv4 shares as NFSv4?
How do I troubleshoot NFS? There is no information in the syslog on neither client nor server.
What are any errors in my configuration?
Facts:

Server is corvina(10.0.0.51)
Client is malbec(10.0.0.1)

Malbec runs Ubuntu 12.04
Server runs Debian 7 wheezy
Both are connected through 1 GbE LAN.
Firewalls are off.
rpcinfo
(root@malbec) (13-07-02 21:00) (P:0 L:1) [0]
~ # rpcinfo -p 
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp   4000  status
    100024    1   tcp   4000  status

(root@malbec) (13-07-02 21:00) (P:0 L:1) [0]
~ # rpcinfo -p corvina 
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp   4000  status
    100024    1   tcp   4000  status
    100003    3   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp   4003  nlockmgr
    100021    3   udp   4003  nlockmgr
    100021    4   udp   4003  nlockmgr
    100021    1   tcp   4003  nlockmgr
    100021    3   tcp   4003  nlockmgr
    100021    4   tcp   4003  nlockmgr
    100005    1   udp   4002  mountd
    100005    1   tcp   4002  mountd
    100005    2   udp   4002  mountd
    100005    2   tcp   4002  mountd
    100005    3   udp   4002  mountd
    100005    3   tcp   4002  mountd

tcpdump
The following is output from tcpdump on malbec while running this command: # rpcinfo -p corvina
~ # tcpdump -i eth0 host 10.0.0.51 
21:14:51.762083 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [S], seq 3069120722, win 14600, options [mss 1460,sackOK,TS val 146111 ecr 0,nop,wscale 7], length 0
21:14:51.762431 IP corvina.vineyard.sikkerhed.org.sunrpc > malbec.vineyard.sikkerhed.org.948: Flags [S.], seq 770684199, ack 3069120723, win 14480, options [mss 1460,sackOK,TS val 398850 ecr 146111,nop,wscale 7], length 0
21:14:51.762458 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [.], ack 1, win 115, options [nop,nop,TS val 146111 ecr 398850], length 0
21:14:51.762556 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [P.], seq 1:45, ack 1, win 115, options [nop,nop,TS val 146111 ecr 398850], length 44
21:14:51.762710 IP corvina.vineyard.sikkerhed.org.sunrpc > malbec.vineyard.sikkerhed.org.948: Flags [.], ack 45, win 114, options [nop,nop,TS val 398850 ecr 146111], length 0
21:14:51.763282 IP corvina.vineyard.sikkerhed.org.sunrpc > malbec.vineyard.sikkerhed.org.948: Flags [P.], seq 1:473, ack 45, win 114, options [nop,nop,TS val 398850 ecr 146111], length 472
21:14:51.763302 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [.], ack 473, win 123, options [nop,nop,TS val 146111 ecr 398850], length 0
21:14:51.764059 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [F.], seq 45, ack 473, win 123, options [nop,nop,TS val 146111 ecr 398850], length 0
21:14:51.764454 IP corvina.vineyard.sikkerhed.org.sunrpc > malbec.vineyard.sikkerhed.org.948: Flags [F.], seq 473, ack 46, win 114, options [nop,nop,TS val 398850 ecr 146111], length 0
21:14:51.764478 IP malbec.vineyard.sikkerhed.org.948 > corvina.vineyard.sikkerhed.org.sunrpc: Flags [.], ack 474, win 123, options [nop,nop,TS val 146111 ecr 398850], length 0

The following is output from tcpdump on malbec while runing this command: ~ # time mount.nfs4 10.0.0.51:/ /mnt -o soft,intr,timeo=10
21:14:58.397327 IP malbec.vineyard.sikkerhed.org.872 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1298959870, win 14600, options [mss 1460,sackOK,TS val 147769 ecr 0,nop,wscale 7], length 0
21:14:58.397655 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.872: Flags [R.], seq 0, ack 1298959871, win 0, length 0
21:14:59.470270 IP malbec.vineyard.sikkerhed.org.854 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 4111013041, win 14600, options [mss 1460,sackOK,TS val 148038 ecr 0,nop,wscale 7], length 0
21:14:59.470569 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.854: Flags [R.], seq 0, ack 4111013042, win 0, length 0
21:15:01.506179 IP malbec.vineyard.sikkerhed.org.988 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1642454567, win 14600, options [mss 1460,sackOK,TS val 148547 ecr 0,nop,wscale 7], length 0
21:15:01.506514 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.988: Flags [R.], seq 0, ack 1642454568, win 0, length 0
21:15:05.542216 IP malbec.vineyard.sikkerhed.org.882 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 3844460520, win 14600, options [mss 1460,sackOK,TS val 149556 ecr 0,nop,wscale 7], length 0
21:15:05.542484 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.882: Flags [R.], seq 0, ack 3844460521, win 0, length 0
21:15:13.602228 IP malbec.vineyard.sikkerhed.org.969 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1317773588, win 14600, options [mss 1460,sackOK,TS val 151571 ecr 0,nop,wscale 7], length 0
21:15:13.602527 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.969: Flags [R.], seq 0, ack 1317773589, win 0, length 0
21:15:18.615027 ARP, Request who-has malbec.vineyard.sikkerhed.org tell corvina.vineyard.sikkerhed.org, length 46
21:15:18.615048 ARP, Reply malbec.vineyard.sikkerhed.org is-at cc:52:af:46:af:23 (oui Unknown), length 28
21:15:23.622223 IP malbec.vineyard.sikkerhed.org.1003 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 2896563167, win 14600, options [mss 1460,sackOK,TS val 154076 ecr 0,nop,wscale 7], length 0
21:15:23.622557 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.1003: Flags [R.], seq 0, ack 2896563168, win 0, length 0
21:15:28.629913 ARP, Request who-has corvina.vineyard.sikkerhed.org tell malbec.vineyard.sikkerhed.org, length 28
21:15:28.630223 ARP, Reply corvina.vineyard.sikkerhed.org is-at 00:9c:02:ab:db:54 (oui Unknown), length 46
21:15:33.662200 IP malbec.vineyard.sikkerhed.org.727 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1334644196, win 14600, options [mss 1460,sackOK,TS val 156586 ecr 0,nop,wscale 7], length 0
21:15:33.663657 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.727: Flags [R.], seq 0, ack 1334644197, win 0, length 0
21:15:43.698207 IP malbec.vineyard.sikkerhed.org.rsync > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 688828331, win 14600, options [mss 1460,sackOK,TS val 159095 ecr 0,nop,wscale 7], length 0
21:15:43.698541 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.rsync: Flags [R.], seq 0, ack 688828332, win 0, length 0
21:15:48.707710 ARP, Request who-has malbec.vineyard.sikkerhed.org tell corvina.vineyard.sikkerhed.org, length 46
21:15:48.707726 ARP, Reply malbec.vineyard.sikkerhed.org is-at cc:52:af:46:af:23 (oui Unknown), length 28
21:15:53.738188 IP malbec.vineyard.sikkerhed.org.946 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 2021272456, win 14600, options [mss 1460,sackOK,TS val 161605 ecr 0,nop,wscale 7], length 0
21:15:53.738519 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.946: Flags [R.], seq 0, ack 2021272457, win 0, length 0
21:16:03.806216 IP malbec.vineyard.sikkerhed.org.902 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 3889059201, win 14600, options [mss 1460,sackOK,TS val 164122 ecr 0,nop,wscale 7], length 0
21:16:03.806546 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.902: Flags [R.], seq 0, ack 3889059202, win 0, length 0
21:16:08.821900 ARP, Request who-has corvina.vineyard.sikkerhed.org tell malbec.vineyard.sikkerhed.org, length 28
21:16:08.822172 ARP, Reply corvina.vineyard.sikkerhed.org is-at 00:9c:02:ab:db:54 (oui Unknown), length 46
21:16:13.874209 IP malbec.vineyard.sikkerhed.org.712 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1480927452, win 14600, options [mss 1460,sackOK,TS val 166639 ecr 0,nop,wscale 7], length 0
21:16:13.874553 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.712: Flags [R.], seq 0, ack 1961062188, win 0, length 0
21:16:18.880588 ARP, Request who-has malbec.vineyard.sikkerhed.org tell corvina.vineyard.sikkerhed.org, length 46
21:16:18.880605 ARP, Reply malbec.vineyard.sikkerhed.org is-at cc:52:af:46:af:23 (oui Unknown), length 28
21:16:23.910209 IP malbec.vineyard.sikkerhed.org.758 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1375860626, win 14600, options [mss 1460,sackOK,TS val 169148 ecr 0,nop,wscale 7], length 0
21:16:23.910532 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.758: Flags [R.], seq 0, ack 1375860627, win 0, length 0
21:16:33.982258 IP malbec.vineyard.sikkerhed.org.694 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 1769203987, win 14600, options [mss 1460,sackOK,TS val 171666 ecr 0,nop,wscale 7], length 0
21:16:33.982579 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.694: Flags [R.], seq 0, ack 1769203988, win 0, length 0
21:16:44.026241 IP malbec.vineyard.sikkerhed.org.841 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 530553783, win 14600, options [mss 1460,sackOK,TS val 174177 ecr 0,nop,wscale 7], length 0
21:16:44.026505 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.841: Flags [R.], seq 0, ack 530553784, win 0, length 0
21:16:46.213388 IP malbec.vineyard.sikkerhed.org.43460 > corvina.vineyard.sikkerhed.org.ssh: Flags [P.], seq 64:128, ack 33, win 325, options [nop,nop,TS val 174723 ecr 397437], length 64
21:16:46.213859 IP corvina.vineyard.sikkerhed.org.ssh > malbec.vineyard.sikkerhed.org.43460: Flags [P.], seq 33:65, ack 128, win 199, options [nop,nop,TS val 427466 ecr 174723], length 32
21:16:46.213883 IP malbec.vineyard.sikkerhed.org.43460 > corvina.vineyard.sikkerhed.org.ssh: Flags [.], ack 65, win 325, options [nop,nop,TS val 174723 ecr 427466], length 0
21:16:54.094242 IP malbec.vineyard.sikkerhed.org.kerberos-master > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 2673083337, win 14600, options [mss 1460,sackOK,TS val 176694 ecr 0,nop,wscale 7], length 0
21:16:54.094568 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.kerberos-master: Flags [R.], seq 0, ack 2673083338, win 0, length 0
21:17:04.134227 IP malbec.vineyard.sikkerhed.org.1019 > corvina.vineyard.sikkerhed.org.nfs: Flags [S], seq 2176607713, win 14600, options [mss 1460,sackOK,TS val 179204 ecr 0,nop,wscale 7], length 0
21:17:04.134566 IP corvina.vineyard.sikkerhed.org.nfs > malbec.vineyard.sikkerhed.org.1019: Flags [R.], seq 0, ack 2176607714, win 0, length 0
21:18:46.314021 IP malbec.vineyard.sikkerhed.org.43460 > corvina.vineyard.sikkerhed.org.ssh: Flags [P.], seq 128:192, ack 65, win 325, options [nop,nop,TS val 204749 ecr 427466], length 64
21:18:46.314462 IP corvina.vineyard.sikkerhed.org.ssh > malbec.vineyard.sikkerhed.org.43460: Flags [P.], seq 65:97, ack 192, win 199, options [nop,nop,TS val 457494 ecr 204749], length 32
21:18:46.314482 IP malbec.vineyard.sikkerhed.org.43460 > corvina.vineyard.sikkerhed.org.ssh: Flags [.], ack 97, win 325, options [nop,nop,TS val 204749 ecr 457494], length 0
21:18:51.317908 ARP, Request who-has corvina.vineyard.sikkerhed.org tell malbec.vineyard.sikkerhed.org, length 28
21:18:51.318177 ARP, Reply corvina.vineyard.sikkerhed.org is-at 00:9c:02:ab:db:54 (oui Unknown), length 46

mount command outputs
mount.nfs4: Connection timed out
mount.nfs4 10.0.0.51:/ /mnt -o soft,intr,timeo=10  0,00s user 0,00s system 0% cpu 2:05,80 total

Returncode is 32
Server configuration
I have enabled idmapd by adding NEED_IDMAPD=yes in /etc/default/nfs-common.
Bind mounts in /etc/fstab:
# nfs-audio
/data/audio     /exports/audio  none    bind  0  0
# nfs-clear
/data/clear     /exports/clear  none    bind  0  0
# nfs-video
/data/video     /exports/video  none    bind  0  0

/etc/exports:
/exports      10.0.0.0/255.255.255.0(rw,no_root_squash,no_subtree_check,fsid=0,crossmnt)
/exports/video      10.0.0.0/255.255.255.0(rw,no_root_squash,no_subtree_check,crossmnt)

Output from # ls -al /exports
total 20
drwxr-xr-x  5 root root  4096 Jul  2 14:14 ./
drwxr-xr-x 28 root root  4096 Jul  2 13:46 ../
drwxr-xr-x  7 tdn  audio 4096 Jun  7 11:30 audio/
drwxr-xr-x 11 root root  4096 Jun 29 12:07 clear/
drwxrwx--- 12 tdn  video 4096 Jun  7 09:46 video/

Edit:
Below is the difference between rpc -p corvina and rpc -p malbec
--- /tmp/rpccorvina     2013-07-03 13:31:56.329229967 +0200
+++ /tmp/rpcmalbec      2013-07-03 13:31:30.069567278 +0200
@@ -7,17 +7,9 @@
     100000    2   udp    111  portmapper
     100024    1   udp   4000  status
     100024    1   tcp   4000  status
-    100003    3   udp   2049  nfs
-    100227    3   udp   2049
     100021    1   udp   4003  nlockmgr
     100021    3   udp   4003  nlockmgr
     100021    4   udp   4003  nlockmgr
     100021    1   tcp   4003  nlockmgr
     100021    3   tcp   4003  nlockmgr
     100021    4   tcp   4003  nlockmgr
-    100005    1   udp   4002  mountd
-    100005    1   tcp   4002  mountd
-    100005    2   udp   4002  mountd
-    100005    2   tcp   4002  mountd
-    100005    3   udp   4002  mountd
-    100005    3   tcp   4002  mountd

A:

OK. I think I solved this. It seems that I had a "-T" in RPCNFSDCOUNT in /etc/default/nfs-kernel-server. This disables TCP which is required for NFSv4.
Removing -T and restarting nfs-kernel-server solves the problem.

