From owner-freebsd-fs@FreeBSD.ORG  Tue Sep 13 09:31:48 2011
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9CDCF1065670
	for <freebsd-fs@freebsd.org>; Tue, 13 Sep 2011 09:31:48 +0000 (UTC)
	(envelope-from freebsd-fs@m.gmane.org)
Received: from lo.gmane.org (lo.gmane.org [80.91.229.12])
	by mx1.freebsd.org (Postfix) with ESMTP id 5725F8FC14
	for <freebsd-fs@freebsd.org>; Tue, 13 Sep 2011 09:31:48 +0000 (UTC)
Received: from list by lo.gmane.org with local (Exim 4.69)
	(envelope-from <freebsd-fs@m.gmane.org>) id 1R3PLH-000166-4u
	for freebsd-fs@freebsd.org; Tue, 13 Sep 2011 11:31:47 +0200
Received: from lara.cc.fer.hr ([161.53.72.113])
	by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
	id 1AlnuQ-0007hv-00
	for <freebsd-fs@freebsd.org>; Tue, 13 Sep 2011 11:31:47 +0200
Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian))
	id 1AlnuQ-0007hv-00
	for <freebsd-fs@freebsd.org>; Tue, 13 Sep 2011 11:31:47 +0200
X-Injected-Via-Gmane: http://gmane.org/
To: freebsd-fs@freebsd.org
From: Ivan Voras <ivoras@freebsd.org>
Date: Tue, 13 Sep 2011 11:31:35 +0200
Lines: 23
Message-ID: <j4n7tn$r1u$1@dough.gmane.org>
References: <freemail.20110912210908.38291.2@xmldata09.freemail.hu>
	<alpine.GSO.2.01.1109121428570.12948@freddy.simplesystems.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Complaints-To: usenet@dough.gmane.org
X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
	rv:6.0.1) Gecko/20110907 Thunderbird/6.0.1
In-Reply-To: <alpine.GSO.2.01.1109121428570.12948@freddy.simplesystems.org>
X-Enigmail-Version: 1.1.2
Subject: Re: ZFS-lighttpd2-sendfile, too high IO
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 13 Sep 2011 09:31:48 -0000

On 12/09/2011 21:32, Bob Friesenhahn wrote:
> On Mon, 12 Sep 2011, Laszlo KAROLYI wrote:
> 
>> But does this explain the 4-5mbyte/s reads when having a 15mbit/s
>> network load?
> 
> There are only two viable explanations:
> 
>   o Insuffient caching due to insufficient resources
> 
>   o Data is not being cached at all
> 
> Zfs reads whole 128K blocks (or whatever the filesystem blocksize is) at
> a time.  It does not read partial blocks from underlying storage. This
> makes it very expensive to perform many small read accesses if the reads
> are not subsequently cached in the ARC.

Yes!

Which makes it particularly "interesting" if you try to run a database
on it while forgetting to reset the block size to e.g. 8K before the
database is initialized - you get 16x more IO than you expected.