From owner-freebsd-fs@FreeBSD.ORG  Wed May 29 13:49:32 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id 6E8FAED4
 for <freebsd-fs@freebsd.org>; Wed, 29 May 2013 13:49:32 +0000 (UTC)
 (envelope-from happe@nbi.dk)
Received: from mail2.nbi.dk (up.nbi.dk [130.225.212.6])
 by mx1.freebsd.org (Postfix) with ESMTP id 3C101F72
 for <freebsd-fs@freebsd.org>; Wed, 29 May 2013 13:49:31 +0000 (UTC)
Received: from [172.24.6.39] (happe.priv.nbi.dk [172.24.6.39])
 by mail2.nbi.dk (Postfix) with ESMTP id A8C173C016
 for <freebsd-fs@freebsd.org>; Wed, 29 May 2013 15:18:37 +0200 (CEST)
Message-ID: <51A6002C.2040803@nbi.dk>
Date: Wed, 29 May 2013 15:18:36 +0200
From: Hans Henrik Happe <happe@nbi.dk>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130317 Thunderbird/17.0.4
MIME-Version: 1.0
To: freebsd-fs@freebsd.org
Subject: ZFS 4k aligned space overhead
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 29 May 2013 13:49:32 -0000

Hi,

I've a system with 3TB WD NAS disks for ZFS. I created a 4k aligned 10 
disk RAIDZ2. I noticed the overhead was ~1.4TB for the file system 
(3*8*10^12/2^40 - <free space>). Then I tried with different number of 
disks:

6: 0.2602885812520981
7: 1.1858475767076015
8: 1.149622268974781
9: 0.7288754601031542
10: 1.3953630803152919
11: 2.061850775964558
12: 2.915792594663799
13: 1.5491229379549623
14: 2.056995471008122
15: 2.5648680040612817
16: 3.0727405650541186
17: 3.5806130981072783
18: 0.7912140190601349

the other good configs (6 and 18) is okay, but it seem strange that 10 
has higher spaceoverhead than 18.

I then tried with RAIDZ:

5: 0.19996666628867388
9: 0.39560710452497005
17: 0.8849408514797688

This seems correct. Then RAIDZ2 again but with ashift=9:

6:  0.2602881230413914
10: 0.4523236369714141
18: 0.7912133224308491

This also seems correct. The 6 and 18 results are basically the same for 
ashift 9 and 12.

Is there an explanation to this?

I'm running FreeBSD 9.1-RELEASE-p3.

Cheers,
Hans Henrik Happe