From owner-freebsd-stable@FreeBSD.ORG Mon Jun 28 15:18:37 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26A6B1065670 for ; Mon, 28 Jun 2010 15:18:37 +0000 (UTC) (envelope-from rick@svn.kiwi-computer.com) Received: from svn.kiwi-computer.com (174-20-59-6.mpls.qwest.net [174.20.59.6]) by mx1.freebsd.org (Postfix) with SMTP id BA1AB8FC18 for ; Mon, 28 Jun 2010 15:18:36 +0000 (UTC) Received: (qmail 53424 invoked by uid 2000); 28 Jun 2010 15:18:35 -0000 Date: Mon, 28 Jun 2010 10:18:35 -0500 From: "Rick C. Petty" To: Jeremy Chadwick Message-ID: <20100628151835.GA53315@kay.kiwi-computer.com> References: <20100627221607.GA31646@kay.kiwi-computer.com> <20100628031401.GA45282@kay.kiwi-computer.com> <20100628034741.GA45748@kay.kiwi-computer.com> <20100628045852.GA75380@icarus.home.lan> <20100628142025.GB52174@kay.kiwi-computer.com> <20100628145600.GA92144@icarus.home.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100628145600.GA92144@icarus.home.lan> User-Agent: Mutt/1.4.2.3i Cc: Rick Macklem , freebsd-stable@freebsd.org Subject: Re: Why is NFSv4 so slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd2009@kiwi-computer.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 15:18:37 -0000 On Mon, Jun 28, 2010 at 07:56:00AM -0700, Jeremy Chadwick wrote: > > Three other things to provide output from if you could (you can X out IPs > and MACs too), from both client and server: > > 6) netstat -idn server: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop nfe0 1500 00:22:15:b4:2d:XX 1767890778 0 0 872169302 0 0 0 nfe0 1500 172.XX.XX.0/2 172.XX.XX.4 1767882158 - - 1964274616 - - - lo0 16384 3728 0 0 3728 0 0 0 lo0 16384 (28)00:00:00:00:00:00:fe:80:00:02:00:00:00:00:00:00:00:00:00:00:00:01 3728 0 0 3728 0 0 0 lo0 16384 (28)00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:01 3728 0 0 3728 0 0 0 lo0 16384 127.0.0.0/8 127.0.0.1 3648 - - 3664 - - - client: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop re0 1500 e0:cb:4e:cd:d3:XX 955288523 0 0 696819089 0 0 0 re0 1500 172.XX.XX.0/2 172.XX.XX.2 955279721 - - 696814499 - - - lo0 16384 3148 0 0 3148 0 0 0 lo0 16384 (28)00:00:00:00:00:00:fe:80:00:02:00:00:00:00:00:00:00:00:00:00:00:01 3148 0 0 3148 0 0 0 lo0 16384 (28)00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:01 3148 0 0 3148 0 0 0 lo0 16384 127.0.0.0/8 127.0.0.1 3112 - - 3112 - - - > 7) sysctl hw.pci | grep msi both server and client: hw.pci.honor_msi_blacklist: 1 hw.pci.enable_msix: 1 hw.pci.enable_msi: 1 > 8) Contents of /etc/sysctl.conf server and client: # 4 virtual channels dev.pcm.0.play.vchans=4 # Read modules from /usr/local/modules kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules # Remove those annoying ARP moved messages: net.link.ether.inet.log_arp_movements=0 # 32MB write cache on disk controllers system-wide vfs.hirunningspace=33554432 # Allow users to mount file systems vfs.usermount=1 # misc net.link.tap.user_open=1 net.inet.ip.forwarding=1 compat.linux.osrelease=2.6.16 debug.ddb.textdump.pending=1 # for NFSv4 kern.ipc.maxsockbuf=524288 > > server, immediately after restarting all of nfs scripts (rpcbind > > nfsclient nfsuserd nfsserver mountd nfsd statd lockd nfscbd): > > > > Jun 27 18:04:44 rpcbind: cannot get information for udp6 > > Jun 27 18:04:44 rpcbind: cannot get information for tcp6 > > These two usually indicate you removed IPv6 support from the kernel, > except your ifconfig output (I've remove it) on the server shows you do > have IPv6 support. I've been trying to get these warnings removed for > quite some time (PR kern/96242). They're harmless, but the > inconsistency here is a little weird -- are you explicitly disabling > IPv6 on nfe0? I have WITHOUT_IPV6= in my make.conf on all my machines (or I have problems with jdk1.6) and WITHOUT_INET6= in my src.conf. I'm not sure why the rpcbind/ifconfig binaries have a different concept than the kernel since I always "make buildworld kernel" and keep things in sync with mergemaster when I reboot. I'm building new worlds/kernels now to see if that makes any difference. -- Rick C. Petty