From owner-freebsd-stable@FreeBSD.ORG Sat Mar 12 02:04:20 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97A1A106564A for ; Sat, 12 Mar 2011 02:04:20 +0000 (UTC) (envelope-from telbizov@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 472DF8FC0C for ; Sat, 12 Mar 2011 02:04:20 +0000 (UTC) Received: by qyk35 with SMTP id 35so101990qyk.13 for ; Fri, 11 Mar 2011 18:04:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vZIZP/rAzfn0ScCj0VJsBn0+yzzA/WS4T9xE3wjt78M=; b=jhAUok/jJn+cokPUTsAzlS3DdwD7xSdgrXFHCokrEfD7JtdcNxg7sq5txxB0KOA4O4 9l9KmVAkszoCJjjGwqhba8oCCCOVyxBb+lkJruaL5T0eYLmIOtlPDhIkfs2JyFoDmA4x ID9q8fa1GAcJYWTe4buTZf44AMxfyt6dOO22s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=r8v96rF7+u+1NqNgAFogzRB8TtdJwto4JayYo5uIm48NGxehf4B3xtigFGoNwzeWG1 KrcIDnp/18rp6D7PePvPx70ylk52ifRUGh/vnz2KQwVVfeSXa9SxHfJLNbU0vHHS0V5n rNlcXVRZdB+1ElkaM5iz7E8s0BSgZQ7xnsxcY= MIME-Version: 1.0 Received: by 10.229.234.12 with SMTP id ka12mr8277980qcb.66.1299895459400; Fri, 11 Mar 2011 18:04:19 -0800 (PST) Received: by 10.229.226.10 with HTTP; Fri, 11 Mar 2011 18:04:19 -0800 (PST) In-Reply-To: <7E4869D6-FE68-4746-BA79-2F709EDE9F67@ebureau.com> References: <2D5317E3-D96F-4123-88D5-5AF1BEAD5B9F@xtechllc.com> <7E4869D6-FE68-4746-BA79-2F709EDE9F67@ebureau.com> Date: Fri, 11 Mar 2011 18:04:19 -0800 Message-ID: From: Rumen Telbizov To: Dustin Wenz Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org Subject: Re: LSI SAS2008 performance with mps(4) driver X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2011 02:04:20 -0000 Dustin, Thanks; It's good to know that it's at least possible to make this work in > some instances. > It does. Unfortunately, our SAS2008 controller is integrated with the logic board (a > SuperMicro X8DT6) connected to a SAS-113TQ backplane. Well I forgot to mention that those 9211-8i's that I have were actually used on X8DT6-F motherboard that I purchased specifically so that I use the built-in controller and save one external. Well listen ... the chip on this mother board is SAS2008 and is exactly the same as far as I can tell. I did go ahead and flashed it too with the latest firmware - 9.00 and it did make a difference for me when playing with the SAS expanders. > It's not so much of an expander; there are two breakout cables that go from > the SFF8087 connectors to individual SATA connectors for each drive on the > backplane. Interesting. It is indeed a direct attached backplane then. Nevertheless I wouldn't be surprised if there's another form of incompatibility with some of the direct attached backplanes. > I've spoken with LSI, and (unsurprisingly) they are unable to provide > firmware for chips which are integrated in 3rd party devices. SuperMicro > also doesn't have any particularly recent updates - the newest I could find > was from last August. > I think version 9.00 will be released any moment now. > If this is indeed an issue with the onboard SAS2008 hardware/firmware, I'm > probably going to have to spring for a separate controller card in the short > term. :/ > > - .Dustin > > On Mar 10, 2011, at 10:09 PM, Rumen Telbizov wrote: > > > Hello Dustin, > > > > I've been testing this SAS2008 LSI chip (on a LSI 9211-8i) for the last > month or so and I can say > > that it makes a pretty good HBA but there are indeed a few caveats you > might need to be aware of. > > In support of that - tonight I finished a FreeBSD 8.2-STABLE machine with > 2 x 24 disk chassis (each with > > a 3Gbit expander) = 48 x 2TB SATA RE4-GP disks in 6 x 8disk raidz2 and I > am able to squeeze out 900MB/s > > write and 1200 MB/s read in a sequential (single dd) manner. The limit > here is the backplane speed. > > > > So back to your problem: > > > > 1) What kind of backplane are you using: please specify the exact model. > Is it a SAS expander or direct attached? > > 3Gbit/s or 6Gbit/s? > > 2) What kind of disk controller exactly are you using? More importantly > what kind of firmware does it have? > > > > Those two are very important. In my case it turned out that if I was > connecting SAS2008 chips > > to pretty much every kind of SuperMicro SAS expander backplanes (tried > against 826EL26, 836E1, 846E1) I was > > getting around 200-300MB/s read/write speeds (FreeBSD and Linux). Direct > attached backplanes (826A) worked fine. > > At the end it turned out that it was some sort of a problem with the LSI > firmware (version 8.00 in my case) and I was given > > to try version 9.00 (soon to be released) which completely solved the > problem. Contact LSI support (very high quality) if you want to try this > firmware. > > > > I can't seem to get any better performance than about 250MB/s writes > through the controller. I'm testing with a zpool of six 250MB magnetic SATA > disks, doing a couple of concurrent sequential writes with dd: > > > > dd bs=128k if=/dev/zero of=/datadisk/zero1 & > > dd bs=128k if=/dev/zero of=/datadisk/zero2 & > > > > > > 3) What kind of zpool raid level do you have those disks organized in? > > 4) Running two parallel dd's on the same pool will actually turn the game > into no-so-sequential type and more of a random access. > > Please try the following and paste results here: > > 4.1) dd if=/dev/zero of=/datadisk/zero1 bs=1M count=50000 (only one dd > and use a file size larger than your memory) > > 4.2) Destroy the zpool (if you have no useful data on it of course) and > try dd against each and every disk individually. > > So something like: > > dd if=/dev/zero of=/dev/da0 bs=1M count=50000 > > dd if=/dev/da0 of=/dev/null bs=1M count=50000 > > monitor throughput with gstat -f da0 or I can send you a simple C program > that I wrote which resembles dd but > > prints stats every second. > > > > On a related note I also experienced very slow read speeds (200MB/s) with > the above mentioned configuration and after enabling > > prefetch (I used to set it to disabled as per Jeremy Chadwick's advise) > everything went back to normal - so keep it in mind. > > > > Cheers, > > Rumen Telbizov > > > > -- > > Rumen Telbizov > > http://telbizov.com > > -- Rumen Telbizov http://telbizov.com