Date: Wed, 28 May 2008 02:03:26 -0400 From: Sten Daniel Soersdal <netslists@gmail.com> To: Alexander Goncharov <alexander.gsander@gmail.com> Cc: freebsd-scsi@freebsd.org Subject: Re: Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID) Message-ID: <483CF5AE.3090105@gmail.com> In-Reply-To: <5c29cc10805260541r1f90f516q8ac7d45d2bb15702@mail.gmail.com> References: <5c29cc10805260541r1f90f516q8ac7d45d2bb15702@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alexander Goncharov wrote: > Hello world, I have faced with following issue on my dedicated server: >=20 > 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10 >=20 > FreeBSD 7.0-generic, 64 bit version >=20 >=20 >=20 > IO performance and behavior is very strange: >=20 > 1) No other process are running: >=20 > Memory stat: >=20 > Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free= >=20 > Copy 3GB file first time >=20 > dd if=3D/home/3gb_file of=3D/home/3gb_file2 > 6291456+0 records in > 6291456+0 records out > 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec) >=20 > 20MBS is very poor=E2=80=A6 >=20 > Memory stat now: >=20 > Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Fr= ee >=20 > 2) Copy the same file again: >=20 > dd if=3D/home/3gb_file of=3D/home/3gb_file2 > 6291456+0 records in > 6291456+0 records out > 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec) >=20 > 100MBs =E2=80=93 much better >=20 > Mem: 9048K Active, 5951M Inact, 287M Wired, 32K Cache, 214M Buf, 25G Fr= ee >=20 > Next attempts with this file show 100MBs spped >=20 > 3)Copy other file first time > dd if=3D/home/test2 of=3D/home/test2_2 > 6144000+0 records in > 6144000+0 records out > 3145728000 bytes transferred in 141.870921 secs (22173170 bytes/sec) >=20 > 20MBs again >=20 > Copy the same file again: >=20 > dd if=3D/home/test2 of=3D/home/test2_2 > 6144000+0 records in > 6144000+0 records out > 3145728000 bytes transferred in 29.560267 secs (106417441 bytes/sec) >=20 > 100MBs =E2=80=93 much better. Memory stat: >=20 > Mem: 8964K Active, 12G Inact, 287M Wired, 28K Cache, 214M Buf, 19G Fre= e >=20 > Copy first file again: >=20 > dd if=3D/home/3gb_file of=3D/home/3gb_file2 > 6291456+0 records in > 6291456+0 records out > 3221225472 bytes transferred in 34.310753 secs (93883847 bytes/sec) >=20 > good speed >=20 > .... >=20 > So, first copying of any file is extremely slow, second and all next > attempts are much better. Please help me with this issue. >=20 > Dmesg some info: >=20 >=20 > ioapic0 <Version 1.1> irqs 0-23 on motherboard > kbd1 at kbdmux0 > ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF54= 13) > hptrr: HPT RocketRAID controller driver v1.1 (May 22 2008 00:49:14) > acpi0: <PTLTD RSDT> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x8008-0x800b on acpi0 >=20 >=20 > amr0: <LSILogic MegaRAID 1.53> mem > 0xdc400000-0xdc40ffff,0xdc100000-0xdc13ffff irq 16 at device 14.0 on pc= i140 > amr0: Using 64-bit DMA > amr0: [ITHREAD] > amr0: delete logical drives supported by controller > amr0: <LSILogic MegaRAID SCSI 320-2E> Firmware 514L, BIOS H430, 128MB R= AM I'm not sure if it's relevant but the LSILogic MegaRAID cards i have are = not able to do true RAID 1+0. Instead it mirrors and then concatenates=20 which will leave you relatively poor read/write performance (no=20 striping). To do RAID 1+0 on my controllers i had to do mirror in BIOS=20 and then stripe the mirror's using gstripe. Also once you do RAID you have to pay great attention to block=20 sizes/widths/etc and alignment of such. Improper alignment will ruin any = performance benefits your RAID levels may provide. Without Battery backed cache your write performance will be poor. Also 'dd' defaults to 512 byte blocks (poor testing). Just my $0.2 --=20 Sten Daniel Soersdal
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?483CF5AE.3090105>