From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 13:14:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDC69106566B for ; Fri, 18 Nov 2011 13:14:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9596B8FC13 for ; Fri, 18 Nov 2011 13:14:47 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 4EB0946B3B; Fri, 18 Nov 2011 08:14:47 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id E7A8B8A051; Fri, 18 Nov 2011 08:14:46 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org Date: Fri, 18 Nov 2011 08:14:42 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201111180814.42656.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 18 Nov 2011 08:14:47 -0500 (EST) Cc: Daryl Sayers Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 13:14:47 -0000 On Thursday, November 17, 2011 10:10:27 pm Daryl Sayers wrote: > > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. I don't know if it will help with your performance, but I have some patches to allow the NFS server to cluster writes. You can try www.freebsd.org/~jhb/patches/nfs_server_cluster.patch. I've tested it on 8, but it should probably apply fine to 9. -- John Baldwin