From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 17:38:56 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18BB81065672 for ; Thu, 10 Jul 2008 17:38:56 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.28]) by mx1.freebsd.org (Postfix) with ESMTP id D31678FC29 for ; Thu, 10 Jul 2008 17:38:55 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so957072yxb.13 for ; Thu, 10 Jul 2008 10:38:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=dJvf1f+3kkEFlfhCmf8seyVPTZQ9IgvjfIxD0MkNbik=; b=iNyxvPGVXbZRgxnyDnkHXPkNTF0UOQObGH2BihU2Z/t6OEoca6qdtZ0LY+RM/zzjlb YbxKZQgQWr7YMy3j7y05AgOYiwLCq7jxm/pMvowQoelIPn+4TWL56uQb/AF9LOsfD+wd oW8wtD2/Nut6Rh4Hh1363sv+UNBUskJVtoZG0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=vPf0Ta7n9dQlW/3F+JiaGtO8c6E7eT/mu9q4j1/sPdG3WLophr/zcsNl1eOL68RvTD lYcelFHty06MZwloqWpgliEdXeBMthRkFoSE+yHB0Pm0HRNgN25BDhjtM0Gno5xhuESo G2WopRmqsRuEeayZDIi8u/yIduzKeFePv5pX8= Received: by 10.114.176.1 with SMTP id y1mr11556090wae.118.1215711534586; Thu, 10 Jul 2008 10:38:54 -0700 (PDT) Received: by 10.114.102.10 with HTTP; Thu, 10 Jul 2008 10:38:54 -0700 (PDT) Message-ID: Date: Thu, 10 Jul 2008 10:38:54 -0700 From: "Jeff Mohler" To: juri_mian@yahoo.com In-Reply-To: <223496.96060.qm@web45607.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <223496.96060.qm@web45607.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 17:38:56 -0000 > Two final questions: > > 1. What would _you_ do with 24 1 TB disks and a 24 port 3ware card ? Assume an i386, 4 GB machine, and that fsck is workable because of "newfs -i 131072" --- Im biased. If I had to have 24TB online, id get a netapp. 24TB of data has gotta be worth some real money, so I'd spend it. No..youre not wrong for NOT doing that, but, I wouldnt consider the 24TB nightmare myself, ive been there before. I never full healed from that mess. But if I had to steer you in a specific direction, plan for total failure. HDD's are built to do one thing. Fail. They occasionally hold data, but in reality, theyre just built to fail. Some are designed to fail sooner than others. You have 24 of them racing for failure. Sounds pessimistic, but, good backup & DR strategies are built around this. > 2. What number should I ask my vendor (3ware) to do the rebuild calculations ? You are talking about IOPS/s - I think I should ask them how many IOPS/s the card does when rebuilding a 24 disk raid-6 array, and then combine that with the IOPS/s I see in my normal workload. --- Rebuild calculations are based around how fast the drives are, cheap SATA is about 14ms track to track (longer full seek) and its highly random, and it has to compete with user/system workload. Thats just not possible to state, too many variables. IOPS are not a card issue, its a physical drive issue. 24TB of FCAL would rebuild faster than SATA, for example. The intelligence of the card itself could come into play, in case it is able to use command queuing to the drives/etc...and if the SATA drives fully support it as well. Depending what you want the system to do for users during the rebuild, prioritize the card appropriately. > How do you measure IOPS/s in FreeBSD on a running machine ? --- iostat -x is a pretty good way to measure that, for the most part. Im prepared to hear about different/better ways.