From owner-freebsd-fs@FreeBSD.ORG Wed Jan 14 18:03:37 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2F82F870 for ; Wed, 14 Jan 2015 18:03:37 +0000 (UTC) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E48873DD for ; Wed, 14 Jan 2015 18:03:36 +0000 (UTC) Received: by mail-ob0-f179.google.com with SMTP id nt9so9325156obb.10 for ; Wed, 14 Jan 2015 10:03:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xKQ6lFptTB7LhAhrHdDGd9uOjfjH1jtelXo6gpCjGfc=; b=x0vrXJQv4cxuOPajlLVz14CILUpU/jwPkFlwz1jtpeE4x0F7u8TON59Jb0RAA5hu5U DpY/ish0vC264S+EyB+PiS6/MSpfvrICNDcw5xKLmALTUP6WR6cHSrjaUbXm5owl3GD3 h/E3RGBuw9qhL54gqKF+QgisRYy9Gu/bxcqy07oxlOidwLwINYh6cyhyis9f85U7/W7V pvN+keJCf540BZWBBndlYQ8xE45/e74Cw+OE4rlSSqOn0NjKJyA3ICAglIl5tkssiCKu Bc2mcsOgq+uSrNEEk5pXb/8iVmq8fM+8cVaro1Xx9nO81iNNlJFdUZD76Z5bn/IItR2t fFmg== MIME-Version: 1.0 X-Received: by 10.60.52.132 with SMTP id t4mr3264486oeo.11.1421258616161; Wed, 14 Jan 2015 10:03:36 -0800 (PST) Received: by 10.202.76.71 with HTTP; Wed, 14 Jan 2015 10:03:36 -0800 (PST) In-Reply-To: <6a3129720b4a439994841c28df676cd1@exch2-4.slu.se> References: <6a3129720b4a439994841c28df676cd1@exch2-4.slu.se> Date: Wed, 14 Jan 2015 10:03:36 -0800 Message-ID: Subject: Re: How many ram... From: Freddie Cash To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 18:03:37 -0000 On Wed, Jan 14, 2015 at 9:47 AM, Karli Sj=C3=B6berg = wrote: > Den 14 jan 2015 18:28 skrev Freddie Cash : > > One of them has dedupe enabled (yeah, yeah, we know, we're moving away > from > > it, it's actually the last one with it enabled), > > But what about all of the savings you were benefitting from? Wasn't it > like 10x dedup savings or something, I know I've asked before at the foru= ms > but a person forgets... What's made you change your mind? > =E2=80=8BOriginally, we were getting great disk space savings =E2=80=8Bthat= made it worthwhile (4x was our lowest, I think our highest was around 8x). However, then we started backing up our mail server with millions of tiny files ... and performance tanked (backups wouldn't complete overnight), especially when deleting old snapshots. We moved the mail server off to it's own backups box without dedupe (it's one of the multi-JBOD storage systems as 1 year of daily backups is 46 TB) and performance went back to usable. Then we started getting issues with resilvers taking 3+weeks to replace disks, monthly scrubs just barely completing before the next one starts, and running out of RAM a lot. When hardware died and killed the pool, we rebuilt it without dedupe and things are running much smoother now. We didn't lose any data as we had it replicated off-site. :) We have 4 storage systems running ZFS: - admin site backups using dedupe with 64 GB of RAM and 16 harddrives - school site backups using compression only, with 64 GB of RAM and 24 harddrives - mail server backups using compression only, with 128 GB of RAM and 90 harddrives - offsite backups storage using dedupe, with 128 GB of RAM and 90 harddrives The long-term goal is to have only the off-site backups storage system using dedupe. And to try and keep it at 90 harddrives. To help with that, we'll be getting another off-site backups storage system for the mail server backups, which will remove the bulk of the data out of the deduped pool. =E2=80=8BWhen we started with ZFS, back in the FreeBSD 7 days, 500 GB serve= r-class harddrives were around $100-200 CDN, and anything over 1 TB was out of our price range, so dedupe was worthwhile (we started with server-class drives attached to 3Ware RAID controllers). Then 2 TB desktop-class drives dropped down around the $120 CDN range and we started replacing them (and using LSI SATA controllers). And dedupe started losing it's awesomeness. Now, we get 2 TB drives in bulk for $80 CDN, so there's no point suffering through the pain points that =E2=80=8Bcome with dedupe on ZFS. --=20 Freddie Cash fjwcash@gmail.com