From owner-freebsd-fs@FreeBSD.ORG Sat Apr 6 14:44:30 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 201EE593; Sat, 6 Apr 2013 14:44:30 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id DDD73FC6; Sat, 6 Apr 2013 14:44:29 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id r36EiLOG020559; Sat, 6 Apr 2013 09:44:21 -0500 (CDT) Date: Sat, 6 Apr 2013 09:44:21 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: "Kenneth D. Merry" Subject: Re: NFS File Handle Affinity ported to new NFS server In-Reply-To: <20130404194719.GA79482@nargothrond.kdm.org> Message-ID: References: <20130404194719.GA79482@nargothrond.kdm.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sat, 06 Apr 2013 09:44:21 -0500 (CDT) Cc: fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2013 14:44:30 -0000 On Thu, 4 Apr 2013, Kenneth D. Merry wrote: > Hi folks, > > I have ported the old NFS server's File Handle Affinity (FHA) code so that > it works with both the old and new NFS servers. > > This sped up sequential reads from ZFS very significantly in my test > scenarios. > > e.g. a single stream read off of an 8-drive RAIDZ2 went from about 75MB/sec > to over 200MB/sec. > > And with 7 read streams from 7 Linux clients coming off of a 36-drive > RAID-10, I went from about 700-800MB/sec to 1.7GB/sec. (This is over > 10Gb ethernet, with 2 aggregated 10Gb ports on the server end.) > > The reason for the speedup is that Linux is doing a lot of prefetching, and > those read requests all wound up going to separate threads in the NFS > server. That confused the ZFS read prefetch code, and caused it to throw > away a lot of data. This seems like a huge improvement. It also raises questions for me. I have been complaining about single-file zfs prefetch performance (to slow of a prefetch ramp) by user-space applications on the Illumos zfs list. Does the underlying zfs code actually know about requesting "threads" rather than an open file handle? If multiple threads read in turn from the same user-space open file handle, will the prefetch become confused due to use of multiple reading threads? Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/