From owner-freebsd-fs@FreeBSD.ORG Mon Oct 22 02:42:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 650215A6 for ; Mon, 22 Oct 2012 02:42:13 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id C8EF48FC0A for ; Mon, 22 Oct 2012 02:42:12 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id e12so1678587lag.13 for ; Sun, 21 Oct 2012 19:42:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fkIix7tEhirLiIq7oHcwMY0dkGayYb0FQ9Do63a/stE=; b=FePQCf2U1DmZD6ypwqih8guWkZ5r1VHWKZKDkfWlBusNDxEsH4irPc5XYo94+jRpmr QKo8CfJRdI2e5Z9Bx/Sm160pR+Mmmw5U0bUsgKcir9lJwcK//LnQ3k418b3wCDnVTQ0W bZV8ZHEK0u7PI4gNlyzsoUG4H+bPA/B1RyIEYI6f88e/qJjXOFtLuq8rMKEHOMOoChid ep1tIg7ktn++iKIw4nnmWudHYSdmKUzk/RbK+Rq4H2Q+emqtBNnPq2xWdGKk3NWvYfcE mdMs79hSm9yfDew2Al4te79RBGtREIue+HOUMJZJ0+ssXaBTDYNgAwJnVixIas+/xA/7 b50w== MIME-Version: 1.0 Received: by 10.112.103.7 with SMTP id fs7mr3102441lbb.25.1350873731506; Sun, 21 Oct 2012 19:42:11 -0700 (PDT) Received: by 10.114.24.66 with HTTP; Sun, 21 Oct 2012 19:42:11 -0700 (PDT) Received: by 10.114.24.66 with HTTP; Sun, 21 Oct 2012 19:42:11 -0700 (PDT) In-Reply-To: <1350834848.88577.33.camel@btw.pki2.com> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> <1350744349.88577.10.camel@btw.pki2.com> <1350765093.86715.69.camel@btw.pki2.com> <508322EC.4080700@FreeBSD.org> <1350778257.86715.106.camel@btw.pki2.com> <1350834848.88577.33.camel@btw.pki2.com> Date: Sun, 21 Oct 2012 19:42:11 -0700 Message-ID: Subject: Re: ZFS HBAs + LSI chip sets (Was: ZFS hang (system #2)) From: Freddie Cash To: Dennis Glatting Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 02:42:13 -0000 On Oct 21, 2012 8:54 AM, "Dennis Glatting" wrote: > > On Sat, 2012-10-20 at 23:52 -0700, Freddie Cash wrote: > > On Oct 20, 2012 5:11 PM, "Dennis Glatting" wrote: > > > > > > > > > I chosen the LSI2008 chip set because the code was donated by LSI, and > > > they therefore demonstrated interest in supporting their products under > > > FreeBSD, and that chip set is found in a lot of places, notably > > > Supermicro boards. Additionally, there were stories of success on the > > > lists for several boards. That said, I have received private email from > > > others expressing frustration with ZFS and the "hang" problems, which I > > > believe are also the LSI chips. > > > > > > I have two questions for the broader list: > > > > > > 1) What HBAs are you using for ZFS and what is your level > > > of success/stability? Also, what is your load? > > > > SuperMicro AOC-USAS-8i using the mpt(4) driver on FreeBSD 9-STABLE in one > > server (alpha). > > > > SuperMicro AOC-USAS2-8i using the mps(4) driver on FreeBSD 9-STABLE in 2 > > servers (beta and omega). > > > > I think they were updated on Oct 10ish. > > > > The alpha box runs 12 parallel rsync processes to backup 50-odd Linux > > servers across multiple data centres. > > > > The beta box runs 12 parallel rsync processes to backup 100-odd Linux and > > FreeBSD servers across 50-odd buildings. > > > > Both boxes uses zfs send to replicate the data to omega (each box saturates > > a 1 Gbps link during the zfs send). > > > > Alpha and omega have 24 SATA 3 Gbps harddrives, configured as 3x 8-drive > > raidz2 vdevs, with a 32 GB SSD split between OS, log vdev, and cache vdev. > > > > Beta has 16 SATA 6 Gbps harddrives, configured into 3x 5-drive raidz2 > > vdevs, with a cold-spare, and a 32 GB SSD split between OS, log vdev, and > > cache vdev. > > > > All three have been patched to support feature flags. All three have > > dedupe enabled, compression enabled, and HPN SSH patches with the NONE > > cipher enabled. > > > > All three run without any serious issues. The only issues we've had are 3, > > maybe 4, situations where I've tried to destroy multi-TB filesystems > > without enough RAM in the machine. We're now running a minimum of 32 GB of > > RAM with 64 GB in one box. > > > > > 2) How well is the LSI chip sets supported under FreeBSD? > > > > I have no complaints. And we're ordering a bunch of LSI 9200-series > > controllers for new servers (PCI brackets instead of UIO). > > > Perhaps I am doing something fundamentally wrong with my SSDs. Currently > I simply add them to a pool after being ashift aligned via gnop (e.g., > -S 4096, depending on page size). > > I remember reading somewhere about offsets to insure data is page > aligned but, IIRC, this was strictly a performance issue. Are you doing > something different? All my harddisks are partitioned the same: # gpart create -s gpt daX # gpart add -b 2048 -t freebsd-zfs -l some-label daX For the SSDs, the above are followed by multiple partitions that are on MB boundaries.