From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 07:18:56 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 103AF158 for ; Wed, 5 Feb 2014 07:18:56 +0000 (UTC) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 741B31760 for ; Wed, 5 Feb 2014 07:18:54 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id s156mS7Y061503 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 5 Feb 2014 08:48:28 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <52F1DEBC.9020304@digsys.bg> Date: Wed, 05 Feb 2014 08:48:28 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: practical maximum number of drives References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> In-Reply-To: <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 07:18:56 -0000 I also wonder how you managed to go over the LSI2008's limit of 112=20 drives... On 05.02.14 07:36, aurfalien wrote: > Hi Graham, > > When you say behaved better with 1 HBA, what were the issues that made = you go that route? > > Also, curious that you have that many drives on 1 PCI card, is it PCI 3= etc=85 and is saturation an issue? > > - aurf > > On Feb 4, 2014, at 8:27 PM, Graham Allan wrote:= > >> This may well be a question with no real answer but since we're specci= ng out a new ZFS-based storage system, I've been asked what the maximum n= umber of drives it can support would be (for a hypothetical expansion opt= ion). While there are some obvious limits such as SAS addressing, I assum= e there must be more fundamental ones in the kernel or drivers, and the p= ractical limits will be very different from the hypothetical ones. >> >> So far the largest system we've built is using three 45-drive chassis = on one SAS2008 (mps) controller, so 135 drives total. Over many months of= running we had several drives fail and be replaced, and eventually the O= S (9.1) failed to assign new da devices. It was time to patch the system = and reboot anyway, which solved it, but we did wonder if we were running = into some kind of limit around 150 drives - though I don't see why. >> >> Interestingly we initially built this system with each drive chassis o= n its own SAS2008 HBA, but it ultimately behaved better daisy-chained wit= h only one. I think I saw a hint somewhere this could be to do with inter= rupt sharing... >> >> Thanks for any insights, >> >> Graham >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"