From owner-freebsd-fs@freebsd.org Sun Apr 10 15:22:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44FD9B0B58D for ; Sun, 10 Apr 2016 15:22:13 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 29B73182E for ; Sun, 10 Apr 2016 15:22:12 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from [172.56.28.49] (port=45007 helo=[172.20.10.4]) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:ECDHE-RSA-AES256-SHA:256) (Exim 4.86_1) (envelope-from ) id 1apGWz-001hwV-VS; Sun, 10 Apr 2016 09:40:08 -0500 Subject: Re: ZFS pool with a large number of filesystems Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: text/plain; charset=utf-8 From: PK1048 In-Reply-To: <34DB45E8-7E1F-4D7C-96FF-E0A403EE8000@omnigroup.com> Date: Sun, 10 Apr 2016 10:39:54 -0400 Cc: "freebsd-fs@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <34DB45E8-7E1F-4D7C-96FF-E0A403EE8000@omnigroup.com> To: Wim Lewis X-Mailer: Apple Mail (2.3124) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Authenticated-Sender: cpanel61.fastdnsservers.com: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Apr 2016 15:22:13 -0000 > On Apr 4, 2016, at 20:38, Wim Lewis wrote: >=20 > I'm curious how many ZFS filesystems are reasonable to have on a = single machine (in a single zpool). We're contemplating a design in = which we'd have tens of thousands, perhaps a couple hundred thousand, = filesystems mounted out of the same pool. Before we go too far into = investigating this idea: Does anyone have real-world experience doing = something like that? Is it a situation that ZFS-on-FreeBSD is engineered = to handle with good performance? Is there a rough estimate of the = resources consumed per additional filesystem (in terms of kernel VM and = disk space)? >=20 > Thanks for any insight or advice (even, or especially, if the answer = is "that's crazy, don't do that" :) ) In a previous life I ran ZFS on Solaris (10) with hundreds of datasets, = but all off of the top level zpool. We also had hundreds of thousands of = snapshots. There was no noticeable hit to user performance, but as = others have pointed out, getting a listing of all the datasets and = snapshots was not a fast operation. zfs list -t filesystem on the other = hand was fast. Fast forwarding to today=E2=80=A6 I have a handful of FreeBSD boxes = running ZFS with less than 100 datasets but with _hourly_ snapshots kept = for 30 days. That is 720 snapshots per dataset. Our largest server has = about 50 datasets, so about 36,000 snapshots. zfs list -t filesystem is = fast while zfs list -t snapshot is slow, but I don=E2=80=99t do that = very often. No performance hit to normal I/O. The FreeBSD hardware is Xeon X55xx or E55xx, usually 2 quad core CPUs, = so 8 or 16 vCPU. RAM varies from low of 12GB to a high of 80GB. Systems = are running a mix of vBox VMs, file service, backup service (including = Mac OS Timemachine backups via NetATalk).