From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 18:42:44 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3897025D for ; Wed, 5 Feb 2014 18:42:44 +0000 (UTC) Received: from mail-pb0-x22a.google.com (mail-pb0-x22a.google.com [IPv6:2607:f8b0:400e:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EFDE51A46 for ; Wed, 5 Feb 2014 18:42:43 +0000 (UTC) Received: by mail-pb0-f42.google.com with SMTP id jt11so721828pbb.15 for ; Wed, 05 Feb 2014 10:42:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=jEl0uVoyep1ObWtTcGjCFgrwjUVD+GdS5RJalLxadD8=; b=JkG/wjITSF+7kNbwaRoHVPJzaDjHb8xOqvjmkBRFrD5ZixDJd1BGOmECI6KWY8Tb4f bL8Ux/1AD6evXhX4WguPJuQ8EJpKxn8PSGhtf0+EcpoS6ofNKCt9scIunbNPRyW3r8OW v5uA5Ol7pYgeC5WRkRvS5wIc6cXLkjDO6rOP1e3vO0vyoKClVEzBou+456XBA6SW5V8t yUXlt0Vwwz2RA2RFefJxkDZZic5vcYRCcB9A5wv6LPDJ8ulZ2ZUbY9SVllRmKLJf1AMU VoLNpEopGEO+ajOj0OVZRgStt7sOj1s4+M4V2Ns6gneoGiDgmmltB1l+OIOhmFTzp1Jb oj5w== X-Received: by 10.69.0.39 with SMTP id av7mr4323797pbd.4.1391625762561; Wed, 05 Feb 2014 10:42:42 -0800 (PST) Received: from briankrusicw.logan.tv ([64.17.255.138]) by mx.google.com with ESMTPSA id ja8sm38456191pbd.3.2014.02.05.10.42.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Feb 2014 10:42:41 -0800 (PST) Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: practical maximum number of drives From: aurfalien In-Reply-To: <52F1FBBA.1000909@digsys.bg> Date: Wed, 5 Feb 2014 10:42:29 -0800 Message-Id: <16004B1D-B8A3-4238-9945-5DA98FCEA254@gmail.com> References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F1DEBC.9020304@digsys.bg> <52F1FBBA.1000909@digsys.bg> To: Daniel Kalchev X-Mailer: Apple Mail (2.1827) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 18:42:44 -0000 Cool. But I was more curious about what lead you to using 1 HBA over using a = few more. You mentioned something about interrupts, what problems manifested as a = result of multi HBAs? - aurf On Feb 5, 2014, at 12:52 AM, Daniel Kalchev wrote: > Ok, two things. >=20 > First, it was a typo -- the number is 122 devices and I actually got = it from the likes of this FAQ entry: = http://www.supermicro.com/support/faqs/faq.cfm?faq=3D10004 > I never use these for anything other than HBA. >=20 > It is interesting to see that LSI claims 3000 devices. Might be, = firmware has changed? Or there are different variations of the = chip/implementation? >=20 > Daniel >=20 > On 05.02.14 10:08, Rich wrote: >> The SAS2008 has a limit of 112 drives? >>=20 >> = http://www.lsi.com/downloads/Public/SAS%20ICs/LSISAS2008/SCG_LSISAS2008_PB= _043009.pdf >> claims "up to 3000 devices." >>=20 >> SAS2008 is a PCIe gen 2 x8 chip. >>=20 >> I suspect the bottleneck order would go SAS expander then SAS2008 = then PCIe. >>=20 >> - Rich >>=20 >> On Wed, Feb 5, 2014 at 1:48 AM, Daniel Kalchev = wrote: >>> I also wonder how you managed to go over the LSI2008's limit of 112 >>> drives... >>>=20 >>>=20 >>> On 05.02.14 07:36, aurfalien wrote: >>>> Hi Graham, >>>>=20 >>>> When you say behaved better with 1 HBA, what were the issues that = made you >>>> go that route? >>>>=20 >>>> Also, curious that you have that many drives on 1 PCI card, is it = PCI 3 >>>> etc... and is saturation an issue? >>>>=20 >>>> - aurf >>>>=20 >>>> On Feb 4, 2014, at 8:27 PM, Graham Allan = wrote: >>>>=20 >>>>> This may well be a question with no real answer but since we're = speccing >>>>> out a new ZFS-based storage system, I've been asked what the = maximum number >>>>> of drives it can support would be (for a hypothetical expansion = option). >>>>> While there are some obvious limits such as SAS addressing, I = assume there >>>>> must be more fundamental ones in the kernel or drivers, and the = practical >>>>> limits will be very different from the hypothetical ones. >>>>>=20 >>>>> So far the largest system we've built is using three 45-drive = chassis on >>>>> one SAS2008 (mps) controller, so 135 drives total. Over many = months of >>>>> running we had several drives fail and be replaced, and eventually = the OS >>>>> (9.1) failed to assign new da devices. It was time to patch the = system and >>>>> reboot anyway, which solved it, but we did wonder if we were = running into >>>>> some kind of limit around 150 drives - though I don't see why. >>>>>=20 >>>>> Interestingly we initially built this system with each drive = chassis on >>>>> its own SAS2008 HBA, but it ultimately behaved better = daisy-chained with >>>>> only one. I think I saw a hint somewhere this could be to do with = interrupt >>>>> sharing... >>>>>=20 >>>>> Thanks for any insights, >>>>>=20 >>>>> Graham >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"