From owner-freebsd-questions@FreeBSD.ORG Sat Jun 23 17:03:09 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6387216A400 for ; Sat, 23 Jun 2007 17:03:09 +0000 (UTC) (envelope-from jwm-freebsd-questions@sentinelchicken.net) Received: from smtp02.lnh.mail.rcn.net (smtp02.lnh.mail.rcn.net [207.172.157.102]) by mx1.freebsd.org (Postfix) with ESMTP id 261D313C4BB for ; Sat, 23 Jun 2007 17:03:09 +0000 (UTC) (envelope-from jwm-freebsd-questions@sentinelchicken.net) Received: from mr08.lnh.mail.rcn.net ([207.172.157.28]) by smtp02.lnh.mail.rcn.net with ESMTP; 23 Jun 2007 12:34:18 -0400 Received: from smtp01.lnh.mail.rcn.net (smtp01.lnh.mail.rcn.net [207.172.4.11]) by mr08.lnh.mail.rcn.net (MOS 3.8.3-GA) with ESMTP id IUZ21309; Sat, 23 Jun 2007 12:34:11 -0400 (EDT) Received: from 207-172-209-180.c3-0.bkl-ubr2.sbo-bkl.ma.static.cable.rcn.com (HELO sentinelchicken.net) ([207.172.209.180]) by smtp01.lnh.mail.rcn.net with SMTP; 23 Jun 2007 12:34:08 -0400 Received: (qmail 15076 invoked from network); 23 Jun 2007 16:34:09 -0000 Received: from unknown (HELO crichton.skepsi.net) (10.0.0.101) by aeryn.skepsi.net with SMTP; 23 Jun 2007 16:34:09 -0000 Received: (nullmailer pid 15073 invoked by uid 1000); Sat, 23 Jun 2007 16:34:09 -0000 Date: Sat, 23 Jun 2007 12:34:09 -0400 From: Jason Morgan To: freebsd-questions@freebsd.org Message-ID: <20070623163409.GA14171@sentinelchicken.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.12-2006-07-14 X-Junkmail-Status: score=10/50, host=mr08.lnh.mail.rcn.net X-Junkmail-SD-Raw: score=unknown, refid=str=0001.0A09020A.467D4B87.001B,ss=1,fgs=0, ip=207.172.4.11, so=2006-12-09 10:45:40, dmn=5.1.5/2006-01-31 Subject: NFS Problems/Questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jun 2007 17:03:09 -0000 I've been having some trouble with NFS performance for some time and now that class is out, I've had a bit of time to investigate but I'm stuck. Below are the details of my investigation. Hopefully, someone here can give me some advice. The basic problem is that my NFS performance is very slow. Right now, I am connecting two workstations to a NFS server, which has my home directory, etc, mounted. They are connected over a gigabit network (right now with mtu set to 7000, which is supported by all hardware -- changing it to 1500 has no effect on performance, which is strange). Each system is running 6.2-RELEASE or -STABLE. Each system is also using the following network card: # ifconfig sk0 sk0: flags=8843 mtu 7000 options=b inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 ether 00:17:9a:bb:05:87 media: Ethernet autoselect (1000baseTX ) status: active # dmesg | grep sk skc0: port 0xec00-0xecff mem 0xfdff8000-0xfdffbfff irq 18 at device 10.0 on pci0 skc0: DGE-530T Gigabit Ethernet Adapter rev. (0x9) sk0: on skc0 sk0: Ethernet address: 00:17:9a:XX:XX:XX ## Server /etc/rc.conf settings rpcbind_enable="YES" rpc_lockd_enable="YES" rpc_statd_enable="YES" nfs_server_enable="YES" nfs_server_flags="-u -t -n 12" nfs_bufpackets="32" mountd_flags="-r" ## Client /etc/rc.conf settings nfs_client_enable="YES" nfs_bufpackets="32" nfsiod_enable="YES" nfsiod_flags="-n 6" rpc_lockd_enable="YES" rpc_statd_enable="YES" rpcbind_enable="YES" ## /etc/exports /usr -alldirs,maproot=root client1 client2 For performance benchmarking, I am using dd. Locally from the server, this is a representative result when writing a 1GB file: ## Local write test (for an upper-bound on what to expect). # dd if=/dev/zero of=./nfs.dat bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 19.580184 secs (53552919 bytes/sec) Connecting from a client (both clients get approximately the same results). ## Remote connection (UDP), mounted in /etc/fstab as with flags: ## rw,-U,-3,-r=32768,-w=32768 # dd if=/dev/zero of=./nfs.dat bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 101.151139 secs (10366428 bytes/sec) ## Remote connection (TCP), mounted in /etc/fstab as with flags: ## rw,-T,-3,-r=32768,-w=32768 # dd if=/dev/zero of=./nfs.dat bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 59.668585 secs (17573334 bytes/sec) As can be seen above, TCP is much faster than UPD. I have tried many different mount settings and these are the best results I could get. To test whether or not I have having network issues, I transferred the same nfs.dat file via a http connection and got ~32MB/sec -- almost 2x the speed of the TCP NFS connection. 32MB/sec is about what I would expect given that my fastest write speed is ~50MB/sec. At this point I am stumped. I have tried increasing/changing the number of nfsiod servers as well as nfs_bufpackets. No matter what settings I change, the results are always the same. I get only two errors, first on /var/log/messages on the server I have just begun seeing: Jun 22 21:13:47 crichton routed[666]: sendto(dc1, 224.0.0.2): Operation not permitted Jun 22 21:13:47 crichton routed[666]: sendto(sk0, 224.0.0.2): Operation not permitted Jun 22 21:13:50 crichton routed[666]: sendto(dc1, 224.0.0.2): Operation not permitted Jun 22 21:13:50 crichton routed[666]: sendto(sk0, 224.0.0.2): Operation not permitted This appeared after I added a route; however, I added the route after many of the tests were done. I get the same results now as before the new route. On one of the clients (the one running 6.2-RELEASE-p1), I also get a nasty error: nfs/tcp clnt: Error 60 reading socket, tearing down TCP connection This cropped up last night after I tweaked some settings. They have now been changed back, but I still get this error. The other client is unaffected. I appreciate any help people can provide on tracking down the issues. Sorry about the long email -- just trying to be thorough. Of course, I've searched the Internet and can't find any clear assistence on these issues. Cheers, ~Jason