From owner-freebsd-current@FreeBSD.ORG Wed Jan 28 06:17:16 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B34C106566B for ; Wed, 28 Jan 2009 06:17:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.175]) by mx1.freebsd.org (Postfix) with ESMTP id F17788FC19 for ; Wed, 28 Jan 2009 06:17:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so7266849wfg.7 for ; Tue, 27 Jan 2009 22:17:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=l90MLmUTWj4opWd6FYeFuPmAg7+bltdVnsQzn5dFSus=; b=IkpKO1FpjWzpmaR3+P1Q4NLhRPicSvq2sAt2VZNgUaptukLJBrIpWUtX/BtEgUhp4J qWMphVPxwLsmDJ7mBbOXxqkG3MX65JyBzydG+dYtp9QlFZP/Yrz+5CsIda1cQNLWX99M DDiH79jqXZIw9eVidXCjmj5dhDOn1aCCOhBpY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=S6knDHsFFWJ6r39L9BxFmweFSo3DxYbQYSjKioqdReoqMk7kEs1rk1F1gRT7QdK1Wx 7QboZGmmOPLfbFdzidegzfd+9ooazN+5EdS6JQb452jYCWs2GgV80OU2ZCl/vFDItnZb 3bF55p7xbaOrSffoO/xiF8LT6Ip95u4G60/AQ= MIME-Version: 1.0 Received: by 10.142.43.7 with SMTP id q7mr445072wfq.316.1233123435684; Tue, 27 Jan 2009 22:17:15 -0800 (PST) In-Reply-To: References: <01N4NEOEB7LY00EQWX@tmk.com> <200901271142.41296.fjwcash@gmail.com> Date: Tue, 27 Jan 2009 22:17:15 -0800 Message-ID: From: Freddie Cash To: freebsd-current@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: Help me select hardware....Some real world data that might help X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jan 2009 06:17:16 -0000 On Tue, Jan 27, 2009 at 8:54 PM, Paul Tice wrote: > I just bumped up the kmem, arc.max, enabled zil and reenabled mdcomp. > Prefetch is disabled. > Less than 1 minute into a backup run of only 4 machines, I've got a fresh > ZFS wedgie. Ouch. Others more knowledgeable than I will have to jump in here, but getting the kmem_max and arc_max settings set right for each system is a bit of an art unto itself. It took almost 6 weeks to get the settings we are currently using, which keeps the system stable while running backups for 83 servers via rsync+ssh. Others find smaller arc_max settings work better, others find smaller kmem_max work better. Each system is different. > I'm intrigued by the ssh/rsync double process throughput bump, but it does > require ssh as well as rsync. Yes, if you aren't using ssh as the transport, then adding -z to rsync can be benfecial. :) > I'm not 100% sure about the gstat, I did no slicing/labeling on the disks, > they are purely /dev/adX used by ZFS, would GEOM level even see this? Yes. All disk access goes through GEOM. You can't hit the device nodes without going through GEOM. It's quite interesting watching the output of gstat. > I suspected you could upsize raidz pools one disk at a time, I do wonder how > the inner/outer track speed differences would affect throughput over a > one-by-one disk full array replacement. And yes, I might wonder too much > about corner cases. ;> We haven't got that far yet. :) We've only filled 6 TB of the 9, so won't be needing to move to larger drives for several more months. -- Freddie Cash fjwcash@gmail.com