Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 30 Mar 2012 17:36:50 -0600
From:      Josh Beard <josh@signalboxes.net>
To:        freebsd-net@freebsd.org
Subject:   NFS:  rpc.statd/lockd becomes unresponsive
Message-ID:  <4F764392.6090400@signalboxes.net>

next in thread | raw e-mail | index | archive | help
Hello,

We've recently setup a FreeBSD 9.0-RELEASE (x64) system to test as an 
NFS server for "live" network homes for Mac clients (mostly 10.5 and 
10.6 clients).

We're a public school district and normally have around 150-200 users 
logged in at a time with network homes.  Currently, we're using netatalk 
(AFP) on a Linux box, after migrating from an aging Mac OS X server.  
Unfortunately, netatalk has some serious performance issues under the 
load we're putting it under and we'd like to migrate to NFS.

We've tried several Linux distributions and various kernels and we're 
now testing FreeBSD (and tested FreeNAS) with similar setups.  
Unfortunately, they all suffer the same issue.

As a test, I have a series of scripts to simulate user activity on the 
clients (e.g. opening Word, opening a browser, doing some read/writes 
with dd, etc).  After a while, NFS on the server runs into an issue 
where (what I think happens) rpc.statd can't talk to rpc.lockd.  Being 
Mac clients, they all get a rather ugly dialog box stating that their 
connection to the server has been lost.

It's worth mentioning that this server is a KVM 'guest' on a Linux 
server.  I'm aware of some I/O issues there, but I don't have a decent 
piece of hardware to really test this on.  I allocated 4 CPUs to it and 
10GB of RAM.  I've tested with the virtio net drivers and without.  
Considering I've seen the same symptoms on around 6 Linux distributions, 
with various kernels, FreeNAS, and FreeBSD, I wouldn't be surprised to 
get the same results if I weren't virtualized.

I haven't really done any tuning on the FreeBSD server, it's fairly vanilla.

We have around ~2600 machines throughout our campus, with limited remote 
management capabilities (that's on the big agenda to tackle), so 
changing NFS mount options there would be rather difficult.  These are 
LDAP accounts with the NFS mounts in LDAP as well, for what it's worth.  
The clients mount it pretty vanilla (output of 'mount' on client):
freenas.dsdk12.schoollocal:/mnt/homes on 
/net/freenas.dsdk12.schoollocal/mnt/homes (nfs, nodev, nosuid, 
automounted, nobrowse)

On the server, my /etc/exports looks like this:
/srv/homes    -alldirs -network    172.30.0.0/16

This export doesn't have a lot of data - it's 150 small home directories 
of test accounts.  No other activity is being done on this server.  The 
filesystem if UFS.

/etc/rc.conf on the server:
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r -l"
nfsd_enable="YES"
mountd_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_flags="-t -n 128"

When this occurs, /var/log/messages starts to fill up with this:

Mar 30 16:35:18 freefs kernel: Failed to contact local NSM - rpc error 5
Mar 30 16:35:20 freefs rpc.statd: unmon request from localhost, no 
matching monitor
Mar 30 16:35:44 freefs rpc.statd: unmon request from localhost, no 
matching monitor
-- repeated a few times every few seconds --
Mar 30 16:54:50 freefs rpc.statd: Unsolicited notification from host 
hs00508s4434.dsdk12.schoollocal
Mar 30 16:55:01 freefs rpc.statd: Unsolicited notification from host 
hs00520s4539.dsdk12.schoollocal
Mar 30 16:55:10 freefs rpc.statd: Failed to call rpc.statd client at 
host localhost

nfsstat shortly after a failure:
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0      1208
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    
Misses
       177       951       226        28         3         6         
0         2
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    
Misses
        49         3        13         5         9         0       
148         9

Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    
Create    Remove
    262698    101012   1575347        29   1924761   2172712         
0     43792
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  
RdirPlus    Access
     27447         0        21      5596      1691    118073         0   
2596146
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0     83638       108       108    183632
Server Ret-Failed
                 0
Server Faults
             0
Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0   9172982
Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
   2172712   2172712         0

rpcinfo shortly after a failure:
    program version netid     address                service    owner
     100000    4    tcp       0.0.0.0.0.111          rpcbind    superuser
     100000    3    tcp       0.0.0.0.0.111          rpcbind    superuser
     100000    2    tcp       0.0.0.0.0.111          rpcbind    superuser
     100000    4    udp       0.0.0.0.0.111          rpcbind    superuser
     100000    3    udp       0.0.0.0.0.111          rpcbind    superuser
     100000    2    udp       0.0.0.0.0.111          rpcbind    superuser
     100000    4    tcp6      ::.0.111               rpcbind    superuser
     100000    3    tcp6      ::.0.111               rpcbind    superuser
     100000    4    udp6      ::.0.111               rpcbind    superuser
     100000    3    udp6      ::.0.111               rpcbind    superuser
     100000    4    local     /var/run/rpcbind.sock  rpcbind    superuser
     100000    3    local     /var/run/rpcbind.sock  rpcbind    superuser
     100000    2    local     /var/run/rpcbind.sock  rpcbind    superuser
     100005    1    udp6      ::.2.119               mountd     superuser
     100005    3    udp6      ::.2.119               mountd     superuser
     100005    1    tcp6      ::.2.119               mountd     superuser
     100005    3    tcp6      ::.2.119               mountd     superuser
     100005    1    udp       0.0.0.0.2.119          mountd     superuser
     100005    3    udp       0.0.0.0.2.119          mountd     superuser
     100005    1    tcp       0.0.0.0.2.119          mountd     superuser
     100005    3    tcp       0.0.0.0.2.119          mountd     superuser
     100024    1    udp6      ::.3.191               status     superuser
     100024    1    tcp6      ::.3.191               status     superuser
     100024    1    udp       0.0.0.0.3.191          status     superuser
     100024    1    tcp       0.0.0.0.3.191          status     superuser
     100003    2    tcp       0.0.0.0.8.1            nfs        superuser
     100003    3    tcp       0.0.0.0.8.1            nfs        superuser
     100003    2    tcp6      ::.8.1                 nfs        superuser
     100003    3    tcp6      ::.8.1                 nfs        superuser
     100021    0    udp6      ::.3.248               nlockmgr   superuser
     100021    0    tcp6      ::.2.220               nlockmgr   superuser
     100021    0    udp       0.0.0.0.3.202          nlockmgr   superuser
     100021    0    tcp       0.0.0.0.2.255          nlockmgr   superuser
     100021    1    udp6      ::.3.248               nlockmgr   superuser
     100021    1    tcp6      ::.2.220               nlockmgr   superuser
     100021    1    udp       0.0.0.0.3.202          nlockmgr   superuser
     100021    1    tcp       0.0.0.0.2.255          nlockmgr   superuser
     100021    3    udp6      ::.3.248               nlockmgr   superuser
     100021    3    tcp6      ::.2.220               nlockmgr   superuser
     100021    3    udp       0.0.0.0.3.202          nlockmgr   superuser
     100021    3    tcp       0.0.0.0.2.255          nlockmgr   superuser
     100021    4    udp6      ::.3.248               nlockmgr   superuser
     100021    4    tcp6      ::.2.220               nlockmgr   superuser
     100021    4    udp       0.0.0.0.3.202          nlockmgr   superuser
     100021    4    tcp       0.0.0.0.2.255          nlockmgr   superuser
     300019    1    tcp       0.0.0.0.2.185          amd        superuser
     300019    1    udp       0.0.0.0.2.162          amd        superuser

The load can get fairly high during my 'stress' tests, but not *that* 
high.  I'm surprised to see these particular symptoms that affect every 
connected user at the same time and would expect slowdowns rather than 
the issue I'm seeing.

Any ideas or nudges in the right direction are most welcome.  This is 
severely plaguing us and our students :\

Thanks,
Josh




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F764392.6090400>