From owner-freebsd-fs@FreeBSD.ORG Tue Jan 22 15:40:29 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B44C7D10 for ; Tue, 22 Jan 2013 15:40:29 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) by mx1.freebsd.org (Postfix) with ESMTP id 7EEB4E38 for ; Tue, 22 Jan 2013 15:40:29 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id k14so11801556iea.24 for ; Tue, 22 Jan 2013 07:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=dCcQIj721dYgKr0N3v/TuQDJYhdhVcBPgVvd1N9AJM4=; b=FUjCvM7HWeAup7n4k/V/p+lGz+FG9ICsmfC3aI/lrUu4Nkx1eCTcAq4g+KL1smOJ52 ENrvt6Ro9VeoVmm6zig0XTQ7SAh8QW5/Vwapg20Ca4ViBdS/pwpkIK2zsXNA9fPgMgc8 iAjVurzYqL8QoLM79fZYAkz6huoOUFE5W+GyY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=dCcQIj721dYgKr0N3v/TuQDJYhdhVcBPgVvd1N9AJM4=; b=HoLfLujbU3Ehb0UaIDf0t63BdAp/H2xtGBu7StZ2B7F5c/iAKUt1Nd7sAHr1vmG7mY RAORK0M7rKoJyi8ByzbyTCoRIW+QfXyRl1tBMdsX5sxSOmksY57UoF7DvoHhhN++cdvQ U38MfCfBUp/KsCjOqJ91lye+49+qSWkWdgZJ8kiSCA14PnMo1IQa3oZh0Xf8CyceCB29 20UMKGHOIteLSPPDrSP6C64gZiRFSMzMpCTsSn5MsrOHbhhbocjXe+61ZiTh3f9I1JEf /FXUNShWKaeelPy3qL7Qbw+i3Dj1yDqMwG0hwK3holWjpR9uHWSPjx4CXFziabTixyr8 x2/w== X-Received: by 10.42.91.7 with SMTP id n7mr14711171icm.40.1358869229011; Tue, 22 Jan 2013 07:40:29 -0800 (PST) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id ez8sm12098977igb.17.2013.01.22.07.40.26 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 22 Jan 2013 07:40:28 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: RFC: Suggesting ZFS "best practices" in FreeBSD From: Kevin Day In-Reply-To: Date: Tue, 22 Jan 2013 09:40:24 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <314B600D-E8E6-4300-B60F-33D5FA5A39CF@sarenet.es> To: Warren Block X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQnl9gnkZTIyzyJDuQA9Jh6xtWfui86+qbHnXFq0EXxGD2BgZ3oTeUsjXrSF40BkpYQPSJ4I Cc: FreeBSD Filesystems , Scott Long , wblock@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2013 15:40:29 -0000 On Jan 22, 2013, at 9:12 AM, Warren Block wrote: >=20 > I would like to see guidelines for at least two common scenarios: >=20 > Multi-terabyte file server with multi-drive pool. [=85] > The first is easy with the defaults, but particular tuning could be = beneficial. And would be a good place to talk about NFS on ZFS, usage = of SSDs, and so on. I run ftpmirror.your.org, which is a 72 x 3TB drive ZFS server. It's a = very busy server. It currently houses the only off-site backup of all of = the Wikimedia projects(121TB), a full FreeBSD FTP mirror(1T), a full = CentOS mirror, all of FreeBSD-Archive(1.5TB), FreeBSD-CVS, etc. It's = usually running between 100 and 1500mbps of ethernet traffic in/out of = it. There are usually around 15 FTP connections, 20-50 HTTP connections, = 10 rsync connections and 1 or 2 CVS connections.=20 The only changes we've made that are ZFS specific are atime=3Doff and = sync=3Ddisabled. Nothing we do uses atimes so disabling that cuts down = on a ton of unnecessary writes. Disabling sync is okay here too - we're = just mirroring stuff that's available elsewhere, so there's no threat of = data loss. Other than some TCP tuning in sysctl.conf, this is running a = totally stock kernel with no special settings.=20 I've looked at using an SSD for meta-data only caching, but it appears = that we've got far more than 256GB of metadata here that's being = accessed regularly (nearly every file is being stat'ed when rsync runs) = so I'm guessing it's not going to be incredibly effective unless I buy a = seriously large SSD. If you have any specific questions I'm happy to answer though. -- Kevin