From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 23:50:59 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF8EC9EA for ; Wed, 2 Jul 2014 23:50:59 +0000 (UTC) Received: from quine.pinyon.org (quine.pinyon.org [65.101.5.249]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A517320AB for ; Wed, 2 Jul 2014 23:50:59 +0000 (UTC) Received: by quine.pinyon.org (Postfix, from userid 122) id D822F1602CD; Wed, 2 Jul 2014 16:50:58 -0700 (MST) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on quine.pinyon.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.0 Received: from feyerabend.n1.pinyon.org (feyerabend.n1.pinyon.org [10.0.10.6]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by quine.pinyon.org (Postfix) with ESMTPSA id 291551602B1 for ; Wed, 2 Jul 2014 16:50:56 -0700 (MST) Message-ID: <53B49AE0.4030902@pinyon.org> Date: Wed, 02 Jul 2014 16:50:56 -0700 From: "Russell L. Carter" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: NFS client READ performance on -current X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2014 23:51:00 -0000 Greetings! It's been 14 years. OMG do I love poudriere and zfs. But apropos of this post from last January: http://lists.freebsd.org/pipermail/freebsd-net/2014-January/037547.html I am going to capitalize READ to emphasize that all I am looking at here is reading on the client. In some of posts I have read people have been confusing read vs. write performance and losing the thread. I have fresh current server and client, each with otherwise quiescent em NICs. I also have debian kernel 3.14-1 server and client. The freebsd hardware is large enough, 16GB RAM + vishera 8 cores. (The debian systems are much smaller/slower) Upshot is that with a -current NFSV4 (4.1) serving a -current client and a linux client, I see today client READ performance of: -current -> -current: 2.7MB/s READ nice and steady -current -> linux: 80MB/s READ very bursty And for the weird (to me) result: linux -> -current: 55.74MB/s READ So to be very clear, the freebsd -current server seems capable of of supporting excellent client READ bandwidth. But something is dramatically awry with the freebsd client. (scp, rsync over ssh, all are near wire speed when I get rid of disk overhead). The -current server has got an 11TB raidz2 behind it but I'm only READing, not writing, so all of the discussion about ZFS + NFS sync write problems seems immaterial, possibly. Here's the mount info: -current client's output from nfsstat -m (terp is current server, ari is linux server): terp:/raid/library on /mnt/terp/library nfsv4,minorversion=1,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 ari:/d1/library on /mnt/ari nfsv4,minorversion=0,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 Linux client's excerpt from mount output: terp:/raid/library on /mnt/terp/library type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.10.3,local_lock=none,addr=10.0.10.4) BTW, linux -> linux is about 80MB/s I tried nfsv3 -current -> -current and still got 2.7MB/s READ. I have spent about 5 hours strolling through the archives of the last several years of discussion about this issue in various forums, but I didn't see a reference once since Jaunary's thread on freebsd-net, and I didn't find a bug entry, so I thought I'd keep it alive, so to speak. I've tried all of the various sysctl tweaks that have been suggested over time, that would make sense for reads, to no effect. Russell