From owner-freebsd-fs@FreeBSD.ORG Mon Jun 16 04:50:57 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B297A97B for ; Mon, 16 Jun 2014 04:50:57 +0000 (UTC) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 812722474 for ; Mon, 16 Jun 2014 04:50:57 +0000 (UTC) Received: by mail-ig0-f175.google.com with SMTP id uq10so2495859igb.2 for ; Sun, 15 Jun 2014 21:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XG+Rri2/4jkK3/oKh2FRZvTjiLXNa4RF+omJ3FvGbWc=; b=SJ5LD0VkwiBu/SvKrGh37jGdO9FaEc+OjpaAeRG+bmkYA74ZATy/JPpiRrp41y5Eb1 mfRUVqREYRrjSWPSuajEOtBwWVIFfrSU0+i6/koLtJVt42ow+TXYf6/1ypad/qpm+mWh La7XHG/6CaSGzTaojOJOasbWgEduHZsXBu4aOlVrI2Q4dI1QY7yHgfb2MQuxadCcLOJm pEe4ILIdC+x2jdCOhCDTrA3ldAlI9BrtLfSzsnbbMd72fszccOCDWWjAZhx5Ptatxdk4 NxyKtZl8HzXzzDINFyvycAoXga6OWi5WBawC7oBCYz+E1u7h3LQkTVeFOnWmSBpfLJaD fBAw== MIME-Version: 1.0 X-Received: by 10.50.1.6 with SMTP id 6mr22917573igi.36.1402894256362; Sun, 15 Jun 2014 21:50:56 -0700 (PDT) Received: by 10.64.225.73 with HTTP; Sun, 15 Jun 2014 21:50:56 -0700 (PDT) In-Reply-To: References: <1402846139.4722.352.camel@btw.pki2.com> Date: Mon, 16 Jun 2014 00:50:56 -0400 Message-ID: Subject: Re: Large ZFS arrays? From: Rich To: Freddie Cash Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2014 04:50:57 -0000 I suppose I should jump in... 8 SC847E26-RJBOD1 per Dell R720 "head" w/128GB RAM, with 2 of the 9201-16e controllers, one port connected per enclosure. (The graphs of how it bottlenecks depending on how you daisy-chain things are...fascinating!) 4 zpools, 11 vdevs of 8 disks each, one disk per JBOD per vdev. SSDs for L2ARC or SLOG are of limited usefulness, given the size of the datasets involved - it'll save you on lots of tiny writes over NFS at times, but otherwise, enough spinning heads will beat the SSDs for sequential IO in non-pathological cases. I can describe more things as desired. :) - Rich On Sun, Jun 15, 2014 at 12:11 PM, Freddie Cash wrote: > On Jun 15, 2014 8:29 AM, "Dennis Glatting" wrote: >> >> Anyone built a large ZFS infrastructures (PB size) and care to share >> words of wisdom? > > We don't yet have a petabyte of storage (currently just under 200 TB raw), > but our infrastructure will scale to 720 TB raw (using 4 TB drives) without > daisy-changing storage boxes, or 1.4 PB if daisy-chained. > > We use a SuperMicro H8DGi-F6 motherboard in an SC826 2U chassis with SSDs > for the OS, log and cache vdevs directly connected to the onboard SAS > controller. We have multiple LSI 9211-8e controllers connected to the > external storage boxes (each chassis has an SAS expander). > > The storage chassis are 45-bay SC846-JBOD chassis, currently using 2TB > drives. We currently only have 2 storage chassis connected. It supports 4 > chassis directly, or 8 if you daisy-chain the storage chassis. > > We currently only use these for backups storage, so we configured things > for bulk storage and not raw I/O or throughout. We only have gigabit > Ethernet, and we saturate that with zfs send every day for several hours. > > Hope that helps. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"