From owner-freebsd-stable@FreeBSD.ORG Fri Mar 11 23:54:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B93A106564A for ; Fri, 11 Mar 2011 23:54:41 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet02.xtechllc.com (internet02.xtechllc.com [65.127.24.21]) by mx1.freebsd.org (Postfix) with ESMTP id 20DA58FC20 for ; Fri, 11 Mar 2011 23:54:40 +0000 (UTC) Received: from service02.office.ebureau.com (service02.office.ebureau.com [192.168.20.15]) by internet02.xtechllc.com (Postfix) with ESMTP id ACE397264E6; Fri, 11 Mar 2011 17:36:50 -0600 (CST) Received: from localhost (localhost [127.0.0.1]) by service02.office.ebureau.com (Postfix) with ESMTP id 8E77B675CAEB; Fri, 11 Mar 2011 17:36:50 -0600 (CST) X-Virus-Scanned: amavisd-new at ebureau.com Received: from service02.office.ebureau.com ([127.0.0.1]) by localhost (service02.office.iscompanies.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D9KiW7yDswaf; Fri, 11 Mar 2011 17:36:49 -0600 (CST) Received: from square.office.iscompanies.com (square.office.iscompanies.com [10.10.20.22]) by service02.office.ebureau.com (Postfix) with ESMTPSA id 6CD80675CAD9; Fri, 11 Mar 2011 17:36:49 -0600 (CST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Dustin Wenz In-Reply-To: Date: Fri, 11 Mar 2011 17:36:49 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <7E4869D6-FE68-4746-BA79-2F709EDE9F67@ebureau.com> References: <2D5317E3-D96F-4123-88D5-5AF1BEAD5B9F@xtechllc.com> To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.1082) Cc: Rumen Telbizov Subject: Re: LSI SAS2008 performance with mps(4) driver X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 23:54:41 -0000 Thanks; It's good to know that it's at least possible to make this work = in some instances. Unfortunately, our SAS2008 controller is integrated with the logic board = (a SuperMicro X8DT6) connected to a SAS-113TQ backplane. It's not so = much of an expander; there are two breakout cables that go from the = SFF8087 connectors to individual SATA connectors for each drive on the = backplane. I've spoken with LSI, and (unsurprisingly) they are unable to = provide firmware for chips which are integrated in 3rd party devices. = SuperMicro also doesn't have any particularly recent updates - the = newest I could find was from last August. If this is indeed an issue with the onboard SAS2008 hardware/firmware, = I'm probably going to have to spring for a separate controller card in = the short term. :/ - .Dustin On Mar 10, 2011, at 10:09 PM, Rumen Telbizov wrote: > Hello Dustin, >=20 > I've been testing this SAS2008 LSI chip (on a LSI 9211-8i) for the = last month or so and I can say > that it makes a pretty good HBA but there are indeed a few caveats you = might need to be aware of. > In support of that - tonight I finished a FreeBSD 8.2-STABLE machine = with 2 x 24 disk chassis (each with > a 3Gbit expander) =3D 48 x 2TB SATA RE4-GP disks in 6 x 8disk raidz2 = and I am able to squeeze out 900MB/s > write and 1200 MB/s read in a sequential (single dd) manner. The limit = here is the backplane speed. >=20 > So back to your problem: >=20 > 1) What kind of backplane are you using: please specify the exact = model. Is it a SAS expander or direct attached? > 3Gbit/s or 6Gbit/s? > 2) What kind of disk controller exactly are you using? More = importantly what kind of firmware does it have? >=20 > Those two are very important. In my case it turned out that if I was = connecting SAS2008 chips > to pretty much every kind of SuperMicro SAS expander backplanes (tried = against 826EL26, 836E1, 846E1) I was > getting around 200-300MB/s read/write speeds (FreeBSD and Linux). = Direct attached backplanes (826A) worked fine. > At the end it turned out that it was some sort of a problem with the = LSI firmware (version 8.00 in my case) and I was given > to try version 9.00 (soon to be released) which completely solved the = problem. Contact LSI support (very high quality) if you want to try this = firmware. >=20 > I can't seem to get any better performance than about 250MB/s writes = through the controller. I'm testing with a zpool of six 250MB magnetic = SATA disks, doing a couple of concurrent sequential writes with dd: >=20 > dd bs=3D128k if=3D/dev/zero of=3D/datadisk/zero1 & > dd bs=3D128k if=3D/dev/zero of=3D/datadisk/zero2 & >=20 >=20 > 3) What kind of zpool raid level do you have those disks organized in? > 4) Running two parallel dd's on the same pool will actually turn the = game into no-so-sequential type and more of a random access. > Please try the following and paste results here: > 4.1) dd if=3D/dev/zero of=3D/datadisk/zero1 bs=3D1M count=3D50000 = (only one dd and use a file size larger than your memory) > 4.2) Destroy the zpool (if you have no useful data on it of course) = and try dd against each and every disk individually. > So something like: > dd if=3D/dev/zero of=3D/dev/da0 bs=3D1M count=3D50000 > dd if=3D/dev/da0 of=3D/dev/null bs=3D1M count=3D50000 > monitor throughput with gstat -f da0 or I can send you a simple C = program that I wrote which resembles dd but=20 > prints stats every second. > =20 > On a related note I also experienced very slow read speeds (200MB/s) = with the above mentioned configuration and after enabling > prefetch (I used to set it to disabled as per Jeremy Chadwick's = advise) everything went back to normal - so keep it in mind.=20 >=20 > Cheers, > Rumen Telbizov >=20 > --=20 > Rumen Telbizov > http://telbizov.com