From owner-freebsd-stable@FreeBSD.ORG Thu Feb 23 20:39:29 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E05F516A420 for ; Thu, 23 Feb 2006 20:39:29 +0000 (GMT) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-c.dmv.com (smtp-gw-cl-c.dmv.com [216.240.97.41]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7EC1243D46 for ; Thu, 23 Feb 2006 20:39:29 +0000 (GMT) (envelope-from sven@dmv.com) Received: from mail-gw-cl-b.dmv.com (mail-gw-cl-b.dmv.com [216.240.97.39]) by smtp-gw-cl-c.dmv.com (8.12.10/8.12.10) with ESMTP id k1NKdSYZ075389 for ; Thu, 23 Feb 2006 15:39:28 -0500 (EST) (envelope-from sven@dmv.com) Received: from lanshark.dmv.com (lanshark.dmv.com [216.240.97.46]) by mail-gw-cl-b.dmv.com (8.12.9/8.12.9) with ESMTP id k1NKdR0u059048 for ; Thu, 23 Feb 2006 15:39:28 -0500 (EST) (envelope-from sven@dmv.com) From: Sven Willenberger To: freebsd-stable@freebsd.org Content-Type: text/plain Date: Thu, 23 Feb 2006 15:41:06 -0500 Message-Id: <1140727266.23965.43.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.4.1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 X-Scanned-By: MIMEDefang 2.48 on 216.240.97.39 Subject: LSI Megaraid (amr) performance woes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2006 20:39:30 -0000 I am having some issues getting any (write) performance out of an LSi Megaraid (320-1) SCSI raid card (using the amr driver). The system is an i386 (p4 xeon) with on-board adaptec scsi controllers and a SUPER GEM318 Saf-te backplane with 6 ea 146GB U320 10k rpm Hitachi drives. dmesg highlights at message end. The main problem I am having is getting anywhere near a decent write performance using the card. I compared having the backplane connected to the on-board adaptec controller to having it connected to the LSi controller. I tried 3 methods of benchmarking. "Adaptec Connected" involved using the on-board adaptec scsi controller, "LSi Connected" involved using the LSI controller as a simple controller having each drive its own logical raid0 drive. LSI write-through and write-back involved using the LSi controller to set up 2 single raid0 drives as their own logical unit and a "spanned" mirror of 4 drives (raid10) as a logical unit (write-back and write-through simply referring to the write method used). In the case of the "XXX Connected" setup, I created a raid10 configuration with 4 of the drives as follows (shown is the commands for adaptec .. for lsi I simply used amrd2 amrd3 etc for the drives). gmirror label -b load md1 da2 da3 gmirror label -b load md2 da4 da5 gmirror load gstripe label -s 65536 md0 /dev/mirror/md1 /dev/mirror/md2 newfs /dev/stripe/md0 mkdir /bench mount /dev/stripe/md0 /bench to test read and write performance I used dd as follows: dd if=/dev/zero of=/raid_or_single_drive/bench64 bs=64k count=32768 which created 2GB files. The summary of results (measured in bytes/sec) is as follows: | SINGLE DRIVE | RAID DRIVE | Connection Method | Write | Read | Write | Read | ------------------------|---------------------|----------------------| adaptec connected | 58808057 | 78188838 | 78625494 | 127331944 | lsi singles | 43944507 | 81238863 | 95104511 | 111626492 | lsi write-through | 45716204 | 81748996 |*10299554*| 108620637 | lsi write-back | 31689131 | 37241934 | 50382152 | 56053085 | With the drives connected to the adaptec controller and using geom, I get the expected increase in write and read performance when moving from a single drive to a raid10 system. Likewise, when using the LSI controller to manage the drives as single units and using geom to create the raid, I get a marked increase in write performance (less of a read increase). However, when using the LSI to create the raid, I end up with a *miserable* 10MB/sec write speed (while achieving acceptable read speeds) in write-through mode and mediocre write speeds in write-back mode (which, without a battery-backed raid card I would rather not do) and, for some reason, a marked decrease in read speeds (over the write-through values). So the question arises as to whether this is an issue with the way the LSI card (320-1) handles "spans" (which I call stripes - versus mirrors) or the way the amr driver views such spans, or an issue with the card not playing nicely with the supermicro motherboard, or perhaps even a defective card. Has anyone else had experience with this card and motherboard combination? As a side note, I also tried dragonfly-bsd (1.4.0) which also uses the amr driver and experienced similar results, and linux (slackware 10.2 default install) which showed write speeds of 45MB/s or so and read speeds of 140MB/s or so using the default LSI controller settings (write-through, 64k stripe size, etc.) Any help or ideas here would be really appreciated in an effort to get anywhere near acceptable write speeds without relying on the unsafe write-back method or excessively sacrificing read speeds. ************************ dmesg highlights: FreeBSD 6.0-RELEASE #0: Thu Nov 3 09:36:13 UTC 2005 root@x64.samsco.home:/usr/obj/usr/src/sys/GENERIC ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Features2=0x4400> Hyperthreading: 2 logical CPUs real memory = 1073217536 (1023 MB) avail memory = 1041264640 (993 MB) pcib5: at device 29.0 on pci4 pci5: on pcib5 amr0: mem 0xfe200000-0xfe20ffff irq 96 at device 1.0 on pci5 amr0: Firmware 1L37, BIOS G119, 64MB RAM pci4: at device 30.0 (no driver attached) pcib6: at device 31.0 on pci4 pci6: on pcib6 ahd0: port 0x4400-0x44ff,0x4000-0x40ff mem 0xfc400000-0xfc401fff irq 76 at device 2.0 on pci6 ahd0: [GIANT-LOCKED] aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs ahd1: port 0x4c00-0x4cff,0x4800-0x48ff mem 0xfc402000-0xfc403fff irq 77 at device 2.1 on pci6 ahd1: [GIANT-LOCKED] aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs Thanks, Sven