From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 09:24:34 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B22694A for ; Wed, 5 Feb 2014 09:24:34 +0000 (UTC) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 606DB1318 for ; Wed, 5 Feb 2014 09:24:34 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id bj1so123262pad.25 for ; Wed, 05 Feb 2014 01:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Eprllk/Rh3NjoCbAiBA5/v7lm/coq/nWeRkx9aFpoFg=; b=QeNDdVGF+3TeP4NxS5ihYKI5AuT0otsjlDENzCC+zkrTIVgikI+Xp+y9JhJOGsRFwa 7wF/lLHkhk2s2q3EIyQvoYq4+YuGDvY9UGAuRbymxM/ybgxSo5Urp+cr2UZ9dtOhVsiG u2XVy/8Bfj4eWc9YPFINZQKYBz70YE3FObZtU9ak4qD5gNh68fZgYCQnGElc1PzMJEMT z/bnHz2Px7f1+aprCeNNla86n7b4CnLYErovWH5Vg+Scsiu085bWTbHZOMP8myPjqOlg ZYCCJe5WnbLxvYcyNua+UNSf4vkrbmRxstyZ6JwK8EDzLJu2cjxdFYiI4j4eAq/+WgBE 6LAw== MIME-Version: 1.0 X-Received: by 10.66.149.231 with SMTP id ud7mr625934pab.8.1391592273432; Wed, 05 Feb 2014 01:24:33 -0800 (PST) Received: by 10.68.126.199 with HTTP; Wed, 5 Feb 2014 01:24:33 -0800 (PST) In-Reply-To: <52F1FBBA.1000909@digsys.bg> References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F1DEBC.9020304@digsys.bg> <52F1FBBA.1000909@digsys.bg> Date: Wed, 5 Feb 2014 04:24:33 -0500 Message-ID: Subject: Re: practical maximum number of drives From: Rich To: Daniel Kalchev Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 09:24:34 -0000 http://www.lsi.com/products/host-bus-adapters/pages/lsi-sas-9200-8e.aspx Claims 512. http://www.lsi.com/products/host-bus-adapters/pages/lsi-sas-9211-4i.aspx claims 256. I'm really extraordinarily curious to see what would happen, but do not have >512 unused drives I could attach to a single HBA to find out...easily. I have >256, but since all the stuff I can find other than SM's documentation claims 512 for the SAS2008, that would only refute their statement, not LSI's. - Rich On Wed, Feb 5, 2014 at 3:52 AM, Daniel Kalchev wrote: > Ok, two things. > > First, it was a typo -- the number is 122 devices and I actually got it from > the likes of this FAQ entry: > http://www.supermicro.com/support/faqs/faq.cfm?faq=10004 > I never use these for anything other than HBA. > > It is interesting to see that LSI claims 3000 devices. Might be, firmware > has changed? Or there are different variations of the chip/implementation? > > Daniel > > > On 05.02.14 10:08, Rich wrote: >> >> The SAS2008 has a limit of 112 drives? >> >> >> http://www.lsi.com/downloads/Public/SAS%20ICs/LSISAS2008/SCG_LSISAS2008_PB_043009.pdf >> claims "up to 3000 devices." >> >> SAS2008 is a PCIe gen 2 x8 chip. >> >> I suspect the bottleneck order would go SAS expander then SAS2008 then >> PCIe. >> >> - Rich >> >> On Wed, Feb 5, 2014 at 1:48 AM, Daniel Kalchev wrote: >>> >>> I also wonder how you managed to go over the LSI2008's limit of 112 >>> drives... >>> >>> >>> On 05.02.14 07:36, aurfalien wrote: >>>> >>>> Hi Graham, >>>> >>>> When you say behaved better with 1 HBA, what were the issues that made >>>> you >>>> go that route? >>>> >>>> Also, curious that you have that many drives on 1 PCI card, is it PCI 3 >>>> etc... and is saturation an issue? >>>> >>>> - aurf >>>> >>>> On Feb 4, 2014, at 8:27 PM, Graham Allan wrote: >>>> >>>>> This may well be a question with no real answer but since we're >>>>> speccing >>>>> out a new ZFS-based storage system, I've been asked what the maximum >>>>> number >>>>> of drives it can support would be (for a hypothetical expansion >>>>> option). >>>>> While there are some obvious limits such as SAS addressing, I assume >>>>> there >>>>> must be more fundamental ones in the kernel or drivers, and the >>>>> practical >>>>> limits will be very different from the hypothetical ones. >>>>> >>>>> So far the largest system we've built is using three 45-drive chassis >>>>> on >>>>> one SAS2008 (mps) controller, so 135 drives total. Over many months of >>>>> running we had several drives fail and be replaced, and eventually the >>>>> OS >>>>> (9.1) failed to assign new da devices. It was time to patch the system >>>>> and >>>>> reboot anyway, which solved it, but we did wonder if we were running >>>>> into >>>>> some kind of limit around 150 drives - though I don't see why. >>>>> >>>>> Interestingly we initially built this system with each drive chassis on >>>>> its own SAS2008 HBA, but it ultimately behaved better daisy-chained >>>>> with >>>>> only one. I think I saw a hint somewhere this could be to do with >>>>> interrupt >>>>> sharing... >>>>> >>>>> Thanks for any insights, >>>>> >>>>> Graham >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>>> >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >>> >>> >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >