Date: Thu, 23 Feb 2006 15:41:06 -0500 From: Sven Willenberger <sven@dmv.com> To: freebsd-stable@freebsd.org Subject: LSI Megaraid (amr) performance woes Message-ID: <1140727266.23965.43.camel@lanshark.dmv.com>
next in thread | raw e-mail | index | archive | help
I am having some issues getting any (write) performance out of an LSi
Megaraid (320-1) SCSI raid card (using the amr driver). The system is an
i386 (p4 xeon) with on-board adaptec scsi controllers and a SUPER GEM318
Saf-te backplane with 6 ea 146GB U320 10k rpm Hitachi drives.
dmesg highlights at message end.
The main problem I am having is getting anywhere near a decent write
performance using the card. I compared having the backplane connected to
the on-board adaptec controller to having it connected to the LSi
controller.
I tried 3 methods of benchmarking. "Adaptec Connected" involved using
the on-board adaptec scsi controller, "LSi Connected" involved using the
LSI controller as a simple controller having each drive its own logical
raid0 drive. LSI write-through and write-back involved using the LSi
controller to set up 2 single raid0 drives as their own logical unit and
a "spanned" mirror of 4 drives (raid10) as a logical unit (write-back
and write-through simply referring to the write method used).
In the case of the "XXX Connected" setup, I created a raid10
configuration with 4 of the drives as follows (shown is the commands for
adaptec .. for lsi I simply used amrd2 amrd3 etc for the drives).
gmirror label -b load md1 da2 da3
gmirror label -b load md2 da4 da5
gmirror load
gstripe label -s 65536 md0 /dev/mirror/md1 /dev/mirror/md2
newfs /dev/stripe/md0
mkdir /bench
mount /dev/stripe/md0 /bench
to test read and write performance I used dd as follows:
dd if=/dev/zero of=/raid_or_single_drive/bench64 bs=64k count=32768
which created 2GB files.
The summary of results (measured in bytes/sec) is as follows:
| SINGLE DRIVE | RAID DRIVE |
Connection Method | Write | Read | Write | Read |
------------------------|---------------------|----------------------|
adaptec connected | 58808057 | 78188838 | 78625494 | 127331944 |
lsi singles | 43944507 | 81238863 | 95104511 | 111626492 |
lsi write-through | 45716204 | 81748996 |*10299554*| 108620637 |
lsi write-back | 31689131 | 37241934 | 50382152 | 56053085 |
With the drives connected to the adaptec controller and using geom, I
get the expected increase in write and read performance when moving from
a single drive to a raid10 system. Likewise, when using the LSI
controller to manage the drives as single units and using geom to create
the raid, I get a marked increase in write performance (less of a read
increase).
However, when using the LSI to create the raid, I end up with a
*miserable* 10MB/sec write speed (while achieving acceptable read
speeds) in write-through mode and mediocre write speeds in write-back
mode (which, without a battery-backed raid card I would rather not do)
and, for some reason, a marked decrease in read speeds (over the
write-through values).
So the question arises as to whether this is an issue with the way the
LSI card (320-1) handles "spans" (which I call stripes - versus mirrors)
or the way the amr driver views such spans, or an issue with the card
not playing nicely with the supermicro motherboard, or perhaps even a
defective card. Has anyone else had experience with this card and
motherboard combination?
As a side note, I also tried dragonfly-bsd (1.4.0) which also uses the
amr driver and experienced similar results, and linux (slackware 10.2
default install) which showed write speeds of 45MB/s or so and read
speeds of 140MB/s or so using the default LSI controller settings
(write-through, 64k stripe size, etc.)
Any help or ideas here would be really appreciated in an effort to get
anywhere near acceptable write speeds without relying on the unsafe
write-back method or excessively sacrificing read speeds.
************************
dmesg highlights:
FreeBSD 6.0-RELEASE #0: Thu Nov 3 09:36:13 UTC 2005
root@x64.samsco.home:/usr/obj/usr/src/sys/GENERIC
ACPI APIC Table: <PTLTD APIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNTX-ID,<b14>>
Hyperthreading: 2 logical CPUs
real memory = 1073217536 (1023 MB)
avail memory = 1041264640 (993 MB)
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci4
pci5: <ACPI PCI bus> on pcib5
amr0: <LSILogic MegaRAID 1.51> mem 0xfe200000-0xfe20ffff irq 96 at
device 1.0 on pci5
amr0: <LSILogic MegaRAID SCSI 320-1> Firmware 1L37, BIOS G119, 64MB RAM
pci4: <base peripheral, interrupt controller> at device 30.0 (no driver
attached)
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4
pci6: <ACPI PCI bus> on pcib6
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0x4400-0x44ff,0x4000-0x40ff mem 0xfc400000-0xfc401fff irq 76 at device
2.0 on pci6
ahd0: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0x4c00-0x4cff,0x4800-0x48ff mem 0xfc402000-0xfc403fff irq 77 at device
2.1 on pci6
ahd1: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
Thanks,
Sven
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1140727266.23965.43.camel>