From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 08:52:15 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 414428D0 for ; Wed, 5 Feb 2014 08:52:15 +0000 (UTC) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AFCE01039 for ; Wed, 5 Feb 2014 08:52:14 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id s158qAEh034786 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 5 Feb 2014 10:52:11 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <52F1FBBA.1000909@digsys.bg> Date: Wed, 05 Feb 2014 10:52:10 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Rich Subject: Re: practical maximum number of drives References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F1DEBC.9020304@digsys.bg> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 08:52:15 -0000 Ok, two things. First, it was a typo -- the number is 122 devices and I actually got it from the likes of this FAQ entry: http://www.supermicro.com/support/faqs/faq.cfm?faq=10004 I never use these for anything other than HBA. It is interesting to see that LSI claims 3000 devices. Might be, firmware has changed? Or there are different variations of the chip/implementation? Daniel On 05.02.14 10:08, Rich wrote: > The SAS2008 has a limit of 112 drives? > > http://www.lsi.com/downloads/Public/SAS%20ICs/LSISAS2008/SCG_LSISAS2008_PB_043009.pdf > claims "up to 3000 devices." > > SAS2008 is a PCIe gen 2 x8 chip. > > I suspect the bottleneck order would go SAS expander then SAS2008 then PCIe. > > - Rich > > On Wed, Feb 5, 2014 at 1:48 AM, Daniel Kalchev wrote: >> I also wonder how you managed to go over the LSI2008's limit of 112 >> drives... >> >> >> On 05.02.14 07:36, aurfalien wrote: >>> Hi Graham, >>> >>> When you say behaved better with 1 HBA, what were the issues that made you >>> go that route? >>> >>> Also, curious that you have that many drives on 1 PCI card, is it PCI 3 >>> etc... and is saturation an issue? >>> >>> - aurf >>> >>> On Feb 4, 2014, at 8:27 PM, Graham Allan wrote: >>> >>>> This may well be a question with no real answer but since we're speccing >>>> out a new ZFS-based storage system, I've been asked what the maximum number >>>> of drives it can support would be (for a hypothetical expansion option). >>>> While there are some obvious limits such as SAS addressing, I assume there >>>> must be more fundamental ones in the kernel or drivers, and the practical >>>> limits will be very different from the hypothetical ones. >>>> >>>> So far the largest system we've built is using three 45-drive chassis on >>>> one SAS2008 (mps) controller, so 135 drives total. Over many months of >>>> running we had several drives fail and be replaced, and eventually the OS >>>> (9.1) failed to assign new da devices. It was time to patch the system and >>>> reboot anyway, which solved it, but we did wonder if we were running into >>>> some kind of limit around 150 drives - though I don't see why. >>>> >>>> Interestingly we initially built this system with each drive chassis on >>>> its own SAS2008 HBA, but it ultimately behaved better daisy-chained with >>>> only one. I think I saw a hint somewhere this could be to do with interrupt >>>> sharing... >>>> >>>> Thanks for any insights, >>>> >>>> Graham >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"