From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 05:36:19 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A4F7741C for ; Wed, 5 Feb 2014 05:36:19 +0000 (UTC) Received: from mail-yh0-x234.google.com (mail-yh0-x234.google.com [IPv6:2607:f8b0:4002:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E8691034 for ; Wed, 5 Feb 2014 05:36:19 +0000 (UTC) Received: by mail-yh0-f52.google.com with SMTP id a41so353627yho.39 for ; Tue, 04 Feb 2014 21:36:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=MrEEiGZvOI1u/kjpQHY1nxYKXANrE1HhkDL6hd5zc8c=; b=xcOrANVsg9V6Hhr8/1+IuUk8PmQOcpqCLOQU4myKHuVVmEBfIM5S4QwnQC6i2g6jrW HtDxmwB+f+K9+huIvb9d0DYV/l5EY3jv1oY+f/jDWS9UDCz3owsvLZ6QYpoMNnFa8eQh VhFEC5eLZQJtAvSm9UJqxA9wGPd+PX1/rf88PbM4cWr4I7/8BXJzFVkqAehAgBgRJbGg QMTOQ2PdSfIkFC6Zxoj3SMUdPmeQlLv+341pBokQqlyFpTJhhvOFWg3WDy8bKp8ua0Wu Xsp8kRasC3UGxwFtDyrXB30cRMJ5PdFl0oA4OtRFSn6iFSW03cdnMLnaQ5shf6nrNXBO Y71g== X-Received: by 10.236.200.35 with SMTP id y23mr41666732yhn.38.1391578578498; Tue, 04 Feb 2014 21:36:18 -0800 (PST) Received: from [192.168.1.76] (75-63-29-182.lightspeed.irvnca.sbcglobal.net. [75.63.29.182]) by mx.google.com with ESMTPSA id w8sm91966081yhg.8.2014.02.04.21.36.17 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Feb 2014 21:36:18 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: practical maximum number of drives From: aurfalien In-Reply-To: <52F1BDA4.6090504@physics.umn.edu> Date: Tue, 4 Feb 2014 21:36:15 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> References: <52F1BDA4.6090504@physics.umn.edu> To: Graham Allan X-Mailer: Apple Mail (2.1827) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 05:36:19 -0000 Hi Graham, When you say behaved better with 1 HBA, what were the issues that made = you go that route? Also, curious that you have that many drives on 1 PCI card, is it PCI 3 = etc=85 and is saturation an issue? - aurf On Feb 4, 2014, at 8:27 PM, Graham Allan wrote: > This may well be a question with no real answer but since we're = speccing out a new ZFS-based storage system, I've been asked what the = maximum number of drives it can support would be (for a hypothetical = expansion option). While there are some obvious limits such as SAS = addressing, I assume there must be more fundamental ones in the kernel = or drivers, and the practical limits will be very different from the = hypothetical ones. >=20 > So far the largest system we've built is using three 45-drive chassis = on one SAS2008 (mps) controller, so 135 drives total. Over many months = of running we had several drives fail and be replaced, and eventually = the OS (9.1) failed to assign new da devices. It was time to patch the = system and reboot anyway, which solved it, but we did wonder if we were = running into some kind of limit around 150 drives - though I don't see = why. >=20 > Interestingly we initially built this system with each drive chassis = on its own SAS2008 HBA, but it ultimately behaved better daisy-chained = with only one. I think I saw a hint somewhere this could be to do with = interrupt sharing... >=20 > Thanks for any insights, >=20 > Graham > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"