From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 08:08:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4DF817D2 for ; Wed, 5 Feb 2014 08:08:37 +0000 (UTC) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 225061B11 for ; Wed, 5 Feb 2014 08:08:37 +0000 (UTC) Received: by mail-pd0-f170.google.com with SMTP id p10so56284pdj.1 for ; Wed, 05 Feb 2014 00:08:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6GrngObxkntJUnB3pd0a62ez/5rR4AlJqZlVapgUWMU=; b=OS7Vq6+lMFRD17rRoS1trVfmecrLFK61S47YzvyZIK5MQJOLyOGGKr7Y/9m0akl45p 0cIhJx/k45duYkE2bA3C67/9AVpSV5FS2Fgj8GzPQ8yTvGjak/pMkxaRPA1R4i9NspKO /wmoUD/IYQCy9JmqLzmGg09D43H7icEc2tUG/T68bTQ281sx+mEGieBL2R7lNnOc/7in iPjkppju5Uzr+4FOy9MnssheZ1KfyiV8OGuGjP7LB0AsHkzVDLVG/eqyAZ4DQlADGWg0 oKqsou8kzkkFzNdMC1XFveo0fIzl/l5sVUgxD8vKsCT06RIOPCJoDMQAa1H4B4Y0l1Ya MGRA== MIME-Version: 1.0 X-Received: by 10.66.163.164 with SMTP id yj4mr54618pab.91.1391587714921; Wed, 05 Feb 2014 00:08:34 -0800 (PST) Received: by 10.68.126.199 with HTTP; Wed, 5 Feb 2014 00:08:34 -0800 (PST) In-Reply-To: <52F1DEBC.9020304@digsys.bg> References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F1DEBC.9020304@digsys.bg> Date: Wed, 5 Feb 2014 03:08:34 -0500 Message-ID: Subject: Re: practical maximum number of drives From: Rich To: Daniel Kalchev Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 08:08:37 -0000 The SAS2008 has a limit of 112 drives? http://www.lsi.com/downloads/Public/SAS%20ICs/LSISAS2008/SCG_LSISAS2008_PB_043009.pdf claims "up to 3000 devices." SAS2008 is a PCIe gen 2 x8 chip. I suspect the bottleneck order would go SAS expander then SAS2008 then PCIe. - Rich On Wed, Feb 5, 2014 at 1:48 AM, Daniel Kalchev wrote: > I also wonder how you managed to go over the LSI2008's limit of 112 > drives... > > > On 05.02.14 07:36, aurfalien wrote: >> >> Hi Graham, >> >> When you say behaved better with 1 HBA, what were the issues that made you >> go that route? >> >> Also, curious that you have that many drives on 1 PCI card, is it PCI 3 >> etc... and is saturation an issue? >> >> - aurf >> >> On Feb 4, 2014, at 8:27 PM, Graham Allan wrote: >> >>> This may well be a question with no real answer but since we're speccing >>> out a new ZFS-based storage system, I've been asked what the maximum number >>> of drives it can support would be (for a hypothetical expansion option). >>> While there are some obvious limits such as SAS addressing, I assume there >>> must be more fundamental ones in the kernel or drivers, and the practical >>> limits will be very different from the hypothetical ones. >>> >>> So far the largest system we've built is using three 45-drive chassis on >>> one SAS2008 (mps) controller, so 135 drives total. Over many months of >>> running we had several drives fail and be replaced, and eventually the OS >>> (9.1) failed to assign new da devices. It was time to patch the system and >>> reboot anyway, which solved it, but we did wonder if we were running into >>> some kind of limit around 150 drives - though I don't see why. >>> >>> Interestingly we initially built this system with each drive chassis on >>> its own SAS2008 HBA, but it ultimately behaved better daisy-chained with >>> only one. I think I saw a hint somewhere this could be to do with interrupt >>> sharing... >>> >>> Thanks for any insights, >>> >>> Graham >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"