From owner-freebsd-fs@FreeBSD.ORG Tue Jun 23 13:32:33 2015 Return-Path: Delivered-To: freebsd-fs@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 004C39BA for ; Tue, 23 Jun 2015 13:32:32 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F6C8BAD for ; Tue, 23 Jun 2015 13:32:31 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id t5NDWO69019797; Tue, 23 Jun 2015 08:32:24 -0500 (CDT) Date: Tue, 23 Jun 2015 08:32:24 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: kpneal@pobox.com cc: FreeBSD FS Subject: Re: ZFS raid write performance? In-Reply-To: <20150623042234.GA66734@neutralgood.org> Message-ID: References: <5587C3FF.9070407@sneakertech.com> <5587C97F.2000407@delphij.net> <55887810.3080301@sneakertech.com> <20150622221422.GA71520@neutralgood.org> <55888E0D.6040704@sneakertech.com> <20150623002854.GB96928@neutralgood.org> <5588D291.4030806@sneakertech.com> <20150623042234.GA66734@neutralgood.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Tue, 23 Jun 2015 08:32:24 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 13:32:33 -0000 On Tue, 23 Jun 2015, kpneal@pobox.com wrote: > > When I was testing read speeds I tarred up a tree that was 700+GB in size > on a server with 64GB of memory. Tar (and cpio) are only single-threaded. They open and read input files one by one. Zfs's read-ahead algorithm ramps up the amount of read-ahead each time the program goes to read data and it is not already in memory. Due to this ramp-up, input file size has a significant impact on the apparent read performance. The ramp-up occurs on a per-file basis. Large files (still much smaller than RAM) will produce a higher data rate than small files. If read requests are pending for several files at once (or several read requests for different parts of the same file), then the observed data rate would be higher. Tar/cpio read tests are often more impacted by disk latencies and zfs read-ahead algorithms than the peak performance of the data path. A very large server with many disks may produce similar timings to a very small server. Long ago I wrote a test script (http://www.simplesystems.org/users/bfriesen/zfs-discuss/zfs-cache-test.ksh) which was intended to expose a zfs bug existing at that time, but is still a very useful test for zfs caching and read-ahead by testing initial sequential read performance from a filesystem. This script was written for Solaris and might need some small adaptation to be used for FreeBSD. Extracting a tar file (particularly on a network client) is a very interesting test of network server write performance. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/