From owner-freebsd-fs@FreeBSD.ORG  Mon Mar 18 19:20:06 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
 by hub.freebsd.org (Postfix) with ESMTP id A02CF71A
 for <freebsd-fs@freebsd.org>; Mon, 18 Mar 2013 19:20:06 +0000 (UTC)
 (envelope-from davide.damico@contactlab.com)
Received: from mail2.shared.smtp.contactlab.it
 (mail2.shared.smtp.contactlab.it [93.94.37.7])
 by mx1.freebsd.org (Postfix) with ESMTP id 08278326
 for <freebsd-fs@freebsd.org>; Mon, 18 Mar 2013 19:20:05 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed;
 q=dns/txt; i=@contactlab.it; t=1363634404;
 h=From:Subject:Date:To:MIME-Version:Content-Type;
 bh=jzrfGO3zwKzlqSZQAxvhI3ql49ohWjs5C8eQDQoj5m8=;
 b=rQ8OGabPXZDmfkoYLM4CFYh6Y9fO4nRFuAflN4WHqqOohhrf7gCT+mhCPYQPqSor
 7B1DZr4QMZZrMY5x9kDCksaOU4ZStYxl6M+6FASaADuN4qZ2aTwQyL1Zx/E/aLfS
 GOjmFzHnldWAGvXNkUATj8FRZ5XMynPYhIj1nZtd3o0=;
Received: from [213.92.90.12] ([213.92.90.12:35596] helo=mail3.tomato.it)
 by t.contactlab.it (envelope-from <davide.damico@contactlab.com>)
 (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP
 id E6/FC-24145-4E867415; Mon, 18 Mar 2013 20:20:04 +0100
Received: from mx3-master.housing.tomato.lan ([172.16.7.55])
 by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD))
 (envelope-from <davide.damico@contactlab.com>) id 1UHfbI-0009p5-J9
 for freebsd-fs@freebsd.org; Mon, 18 Mar 2013 20:20:04 +0100
Received: (qmail 37758 invoked by uid 80); 18 Mar 2013 19:20:04 -0000
To: Steven Hartland <killing@multiplay.co.uk>
Subject: Re: FreBSD 9.1 and ZFS v28 performances
X-PHP-Script: uebmeil.sys.tomatointeractive.it/index.php for 172.16.16.227
X-PHP-Originating-Script: 0:main.inc
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Mon, 18 Mar 2013 20:20:04 +0100
From: Davide D'Amico <davide.damico@contactlab.com>
Organization: ContactLab
Mail-Reply-To: <davide.damico@contactlab.com>
In-Reply-To: <A9C89C979F8D4CBABE73256A1C66E95D@multiplay.co.uk>
References: <514729BD.2000608@contactlab.com>
 <810E5C08C2D149DBAC94E30678234995@multiplay.co.uk>
 <51473D1D.3050306@contactlab.com>
 <1DD6360145924BE0ABF2D0979287F5F4@multiplay.co.uk>
 <51474F2F.5040003@contactlab.com>
 <E106A7DB08744581A08C610BD8A86560@multiplay.co.uk>
 <51475267.1050204@contactlab.com> <514757DD.9030705@contactlab.com>
 <A9C89C979F8D4CBABE73256A1C66E95D@multiplay.co.uk>
Message-ID: <4280fd76b9b18376e11e90705e0c736c@sys.tomatointeractive.it>
X-Sender: davide.damico@contactlab.com
User-Agent: Roundcube Webmail/0.8.5
Cc: freebsd-fs@freebsd.org
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
Reply-To: davide.damico@contactlab.com
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Mar 2013 19:20:06 -0000

[...]

> Oh and another thing if this is mysql did you set the right settings
> for your ZFS volume e.g.
> zfs set atime=off tank
> zfs create tank/mysql
> zfs set recordsize=16k tank/mysql

# zfs get all DATA | egrep -e "atime|record"
DATA  recordsize            16K                    local
DATA  atime                 off                    local

I didn't create tank/mysql, I issued:
# zpool create DATA mirror ...
# cd /DATA
# zfs set atime=off DATA
# zfs set recordsize=16k DATA
# mkdir mysql
# chown mysql:mysql mysql
# cp -Rp /repo/mysql/* /DATA/mysql
[..]
#

I think it's the same.

I've used:
vfs.zfs.prefetch_disable="1"

and
vfs.zfs.prefetch_disable="0"

same result.

I tried enabling/disabling in /boot/loader.conf (I have 32GB ram):
vfs.zfs.arc_min="4096M"
vfs.zfs.arc_max="15872M"
vm.kmem_size_max="64G"
vm.kmem_size="49152M"
vfs.zfs.write_limit_override=1073741824

same result.

Thanks,
d.