From owner-freebsd-fs@FreeBSD.ORG Tue Jul 10 19:48:58 2012 Return-Path: <owner-freebsd-fs@FreeBSD.ORG> Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 122831065687 for <freebsd-fs@freebsd.org>; Tue, 10 Jul 2012 19:48:58 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id B2C688FC0A for <freebsd-fs@freebsd.org>; Tue, 10 Jul 2012 19:48:57 +0000 (UTC) Received: by yenl8 with SMTP id l8so480582yen.13 for <freebsd-fs@freebsd.org>; Tue, 10 Jul 2012 12:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=6izRNJUnPs6Zohq7vCCXihk8tcKll9x5+AT5oVzjbqE=; b=n3OZbE4AWR9MlmcDxdBBr7NPB3EkxkaTrM/nU4OTl76J5qv3BWPqNXOhOBVZ3f4g5L 6BviC4lBiOw+kqxNFGRZhakoHOhVbqeaO9EXcJs5o1QrgXMlXJGT1g7yfgh2zIgdrJEA vgzhx8+ONN86IoAHJdFgWOKT+B5ybc5LZVaSk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=6izRNJUnPs6Zohq7vCCXihk8tcKll9x5+AT5oVzjbqE=; b=TeNFDeYjP8B9T7v26nP7GmxDP4BwM/EI9qi6dhTLGqsQXjglM07le+Bggewol7hvkW lmX/jPjOM52PMZqP4hFlqfWKLNNN1Mu4zK/yh1POu1r/3WLA3ZkrZeIS1EIszHLA+Lcy IGu6taMjoCxKIoWO4Za151W9pczvbhBckPRwwMfLtlj4P2PxDbuuo9pvs4MHG9WTSFjL WReeAIN3oy+Ccv3Ux8sNm/yfrbKSjYqgbH0/qENu+T9tj/MPjbTaSvCwRSRIRRJ+8YGx TVQiqJBr+hzI2cl78XSDokRw30tfslvS4/8wffLsoupixw2aZvEamo97WEHa3RjuZ0GS Lp2g== Received: by 10.43.69.12 with SMTP id ya12mr23712317icb.50.1341949736924; Tue, 10 Jul 2012 12:48:56 -0700 (PDT) Received: from static177.us.your.org (static177.us.your.org. [204.9.55.177]) by mx.google.com with ESMTPS id ay5sm12424400igb.15.2012.07.10.12.48.52 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 10 Jul 2012 12:48:55 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) From: Kevin Day <toasty@dragondata.com> In-Reply-To: <1341946657.18535.YahooMailClassic@web122505.mail.ne1.yahoo.com> Date: Tue, 10 Jul 2012 14:48:51 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <AEA750AD-BF15-45D1-9F78-460FB6A40A40@dragondata.com> References: <1341946657.18535.YahooMailClassic@web122505.mail.ne1.yahoo.com> To: Jason Usher <jusher71@yahoo.com> X-Mailer: Apple Mail (2.1485) X-Gm-Message-State: ALoCoQmzJHb6qbrfw141ZIs6Kwl4UJL8XGKJahn7pued+3U5U7Htan924Nls0A6dFWw+jaL1mlyj Cc: freebsd-fs@freebsd.org Subject: Re: chaining JBOD chassic to server ... why am I scared ? (ZFS) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs> List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 10 Jul 2012 19:48:58 -0000 On Jul 10, 2012, at 1:57 PM, Jason Usher <jusher71@yahoo.com> wrote: > The de-facto configuration the smart folks are using for ZFS seems to = be: >=20 > - 16/24/36 drive supermicro chassis > - LSI 9211-8i internal cards > - ZFS and probably raidz2 or raidz3 vdevs >=20 > Ok, fine. But then I see some even smarter folks attaching the = 48-drive 4U JBOD chassis to this configuration, probably using a = different LSI card that has an external SAS cable. >=20 > So ... 84 drives accessible to ZFS on one system. In terms of space = and money efficiency, it sounds really great - fewer systems to manage, = etc. >=20 > But this scares me ... >=20 > - two different power sources - so the "head unit" can lose power = independent of the JBOD device ... how well does that turn out ? >=20 > - external cabling - has anyone just yanked that external SAS cable a = few times, and what does that look like ? >=20 > - If you have a single SLOG, or a single L2ARC device, where do you = put it ? And then what happens if "the other half" of the system = detaches from the half that the SLOG/L2ARC is in ? >=20 > - ... any number of other weird things ? >=20 >=20 > Just how well does ZFS v28 deal with these kind of situations, and do = I have a good reason to be awfully shy about doing this ? >=20 We do this for ftpmirror.your.org (which is ftp3.us.freebsd.org & = others). It's got an LSI 9280 in it, which has 3 external chassis (each = with 24 3TB drives) attached to it. Before putting into use, we = experimented with pulling the power/data cables from random places while = using it. Nothing we did was any worse than the whole system just losing = power. The only difference was that in some cases losing all the storage = would hang the server until it was power cycled, but again=85 no worse = than if everything lost power. If something goes bad, it's pretty = likely things are going to go down, no matter the physical topology. = There was no crazy data loss or anything if that's what you're worried = about. -- Kevin