From owner-freebsd-fs@FreeBSD.ORG Sun Oct 21 06:52:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 183AFD8A for ; Sun, 21 Oct 2012 06:52:09 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 809278FC12 for ; Sun, 21 Oct 2012 06:52:07 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id e12so1315007lag.13 for ; Sat, 20 Oct 2012 23:52:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ehs75o5+vKHk7/lu4FkdJKG3p5jSNd4RCAxYL8iLtJM=; b=f3zTgKaO2DWa+GtCMJEU5vAmDgo5XGmqvKyQaO8KMvmoowFG8LJoAx2gqS4wcfFv49 hjRKampLwnxKkFiMfgkDCXpWP/KcGrvHISh/qRttqHBX3k9REu617uoSUBG1CSUnuLCr /dgbH6lsf+dkegDvTAsovgMFV0pl05yEgPv+4OT9LwZoX3MoXiHe9re50aCsZKcH651L GgfVLfDIM+9EoBcEUIS6v9wMxZubx6TaM4IHWbxeqsbmoNYu5AM580qCGJfVC12WpxSL jP6gDR9C4A5HFNuBdgSGISOL7IyVlAZh2pvez0PDyFPScxbZLW9R3H60LTtR6eFkaDOT ktfg== MIME-Version: 1.0 Received: by 10.152.105.135 with SMTP id gm7mr5062073lab.22.1350802326255; Sat, 20 Oct 2012 23:52:06 -0700 (PDT) Received: by 10.114.24.66 with HTTP; Sat, 20 Oct 2012 23:52:06 -0700 (PDT) Received: by 10.114.24.66 with HTTP; Sat, 20 Oct 2012 23:52:06 -0700 (PDT) In-Reply-To: <1350778257.86715.106.camel@btw.pki2.com> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> <1350744349.88577.10.camel@btw.pki2.com> <1350765093.86715.69.camel@btw.pki2.com> <508322EC.4080700@FreeBSD.org> <1350778257.86715.106.camel@btw.pki2.com> Date: Sat, 20 Oct 2012 23:52:06 -0700 Message-ID: Subject: Re: ZFS HBAs + LSI chip sets (Was: ZFS hang (system #2)) From: Freddie Cash To: Dennis Glatting Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 06:52:09 -0000 On Oct 20, 2012 5:11 PM, "Dennis Glatting" wrote: > > > I chosen the LSI2008 chip set because the code was donated by LSI, and > they therefore demonstrated interest in supporting their products under > FreeBSD, and that chip set is found in a lot of places, notably > Supermicro boards. Additionally, there were stories of success on the > lists for several boards. That said, I have received private email from > others expressing frustration with ZFS and the "hang" problems, which I > believe are also the LSI chips. > > I have two questions for the broader list: > > 1) What HBAs are you using for ZFS and what is your level > of success/stability? Also, what is your load? SuperMicro AOC-USAS-8i using the mpt(4) driver on FreeBSD 9-STABLE in one server (alpha). SuperMicro AOC-USAS2-8i using the mps(4) driver on FreeBSD 9-STABLE in 2 servers (beta and omega). I think they were updated on Oct 10ish. The alpha box runs 12 parallel rsync processes to backup 50-odd Linux servers across multiple data centres. The beta box runs 12 parallel rsync processes to backup 100-odd Linux and FreeBSD servers across 50-odd buildings. Both boxes uses zfs send to replicate the data to omega (each box saturates a 1 Gbps link during the zfs send). Alpha and omega have 24 SATA 3 Gbps harddrives, configured as 3x 8-drive raidz2 vdevs, with a 32 GB SSD split between OS, log vdev, and cache vdev. Beta has 16 SATA 6 Gbps harddrives, configured into 3x 5-drive raidz2 vdevs, with a cold-spare, and a 32 GB SSD split between OS, log vdev, and cache vdev. All three have been patched to support feature flags. All three have dedupe enabled, compression enabled, and HPN SSH patches with the NONE cipher enabled. All three run without any serious issues. The only issues we've had are 3, maybe 4, situations where I've tried to destroy multi-TB filesystems without enough RAM in the machine. We're now running a minimum of 32 GB of RAM with 64 GB in one box. > 2) How well is the LSI chip sets supported under FreeBSD? I have no complaints. And we're ordering a bunch of LSI 9200-series controllers for new servers (PCI brackets instead of UIO).