From owner-freebsd-fs@freebsd.org Sat Sep 19 17:24:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE3E69CEF3F for ; Sat, 19 Sep 2015 17:24:03 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from douhisi.pair.com (douhisi.pair.com [209.68.5.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F8F31E61 for ; Sat, 19 Sep 2015 17:24:03 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from [10.2.2.1] (pool-173-48-121-235.bstnma.fios.verizon.net [173.48.121.235]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by douhisi.pair.com (Postfix) with ESMTPSA id 4C0EE3F778 for ; Sat, 19 Sep 2015 13:23:56 -0400 (EDT) Message-ID: <55FD9A2B.8060207@sneakertech.com> Date: Sat, 19 Sep 2015 13:23:55 -0400 From: Quartz MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS cpu requirements, with/out compression and/or dedup References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Sep 2015 17:24:03 -0000 > from what i read the status of dedup is not that clear and seems there are > bugs and better to avoid it? There aren't [m]any legitimate bugs with dedup, the problem is that it consumes metric assloads of ram, on the order of 2-5GB per TB of disk space (this is in addition to whatever the ARC eats). If you run out of ram performance jumps off a cliff. It's also 'permanent' in the sense that you have to turn it on with the creation of a dataset and can't disable it without nuking said dataset. The issue of "bugs" are probably from people experiencing file corruption because they enabled only block checksums without also exact verification, and don't understand hash collisions. > so according to 1-3 above what cpu requirements i need? Not much. CPU only really takes a hit when you turn on gzip compression and crank it up to high levels. > supermicro c2750/3/5/8 enough to run system of 20TB /40TB with 1-3 above? > if dedup IS enabled would it still work fine? Probably not. That board only supports up to 64GB of ram. 20TB of disk space with dedup will require like 50-100 GB of ram, and 40TB will need a good 90GB at least.