From owner-freebsd-net@FreeBSD.ORG Sun Jan 26 02:36:33 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64AC9257; Sun, 26 Jan 2014 02:36:33 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 03BF11E4D; Sun, 26 Jan 2014 02:36:32 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ag8UACJ05FKDaFve/2dsb2JhbABagXICAYFPVoJ9tVKDL0+BH3SCJQEBAQMBAQEBICsgCwUWGAICDRkCKQEJJg4HBAEcBIdcCA2rYJwvF4EpjRMBAQ0ONAcWglmBSQSJSIwMhAWQbINLHjF9Bxci X-IronPort-AV: E=Sophos;i="4.95,721,1384318800"; d="scan'208";a="90488526" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 25 Jan 2014 21:36:25 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 026E5B4042; Sat, 25 Jan 2014 21:36:26 -0500 (EST) Date: Sat, 25 Jan 2014 21:36:26 -0500 (EST) From: Rick Macklem To: wollman@freebsd.org Message-ID: <188195924.16327973.1390703786000.JavaMail.root@uoguelph.ca> In-Reply-To: <201401260225.s0Q2PUp1045129@hergotha.csail.mit.edu> Subject: Re: Terrible NFS performance under 9.2-RELEASE? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jan 2014 02:36:33 -0000 Garrett Wollman wrote: > In article > <278396201.16318356.1390701347722.JavaMail.root@uoguelph.ca>, Rick > Macklem writes: > > >Well, when I get home in April, I'll try the fairly recent Linux > >client > >I have at home and see what it does. Not sure what trick they could > >use > >to avoid the read before write for partial pages. (I suppose I can > >look at their sources, but that could be pretty scary;-) > > For what it's worth, our performance for large-block 100%-read > workloads is also not what it could (or ought to) be. Between two > 20G-attached servers, I can get about 12 Gbit/s with three parallel > TCP connections. (Multiple connections are required to trick the > lagg > hash into balancing the load across both 10G links, because the hash > function used for load-balancing uses the source and destination > ports.) On the same pair of servers, "dd if=/mnt/test bs=1024k" runs > at about 3 Gbit/s, whereas reading from the local filesystem goes > anywhere from 1.5 to 3 G*byte*/s (i.e., eight times faster) with much > higher CPU utilization. Luckily, most of our users are only > connected > at 1G so they don't notice. > Have you tried increasing readahead by any chance? I think the default is 1, which means the client will make 2 read requests and then wait for those replies before doing any more reads. Since you have fast links, maybe the 2 * 64K reads isn't enough to keep the pipe filled? (This depends on latency, which you didn't mention.) Might be worth trying, rick ps: If/when you have a test server, you could also try compiling a kernel with MAXBSIZE set to 128Kbytes instead of 64Kbytes. You'll need to boot this kernel on both the server and client (assuming a FreeBSD client) before the default rsize will increase to 128Kbytes. I'm no ZFS guy, but I understand 128Kbytes is the blocksize it likes. > I'm going to lose my test server soon (it has to go into production > shortly), so I'm not really able to work on this. I'll have another > test server soon (old hardware being replaced by the new server) and > hope to be able to try out the new code that's going to be in 10.1, > with the expectation of upgrading to 10.x over summer break. > > -GAWollman > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >