Date: Wed, 11 Nov 1998 11:32:15 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Martin Cracauer <cracauer@cons.org> Cc: =?iso-8859-1?Q?S=F8ren_Schmidt?= <sos@freebsd.dk>, Martin Cracauer <cracauer@cons.org>, john@isi.co.jp, grog@lemis.com, hackers@FreeBSD.ORG, mike@smith.net.au, ticso@cicely.de Subject: Re: SCSI vs. DMA33.. Message-ID: <199811111932.LAA22898@apollo.backplane.com> References: <19981111164136.A29513@cons.org> <199811111730.SAA01133@freebsd.dk> <19981111184459.A10448@cons.org>
next in thread | previous in thread | raw e-mail | index | archive | help
:What means "work"? I get a functional ccd filessystem over 4
:drives/4channels/2cards, so far so good.
:
:But while the throughput is raised when striping over different cards,
:it isn't when striping over two channels on the same card. Or in other
:words, the maximum thoughput for a
:(2x) 4-drive, 4-channel, 2-card ccd is the same as for a
:(2x) 2-drive, 2-channel, 2-card array, while
:(1x) 2-drive, 2-channel, 1-card is excatly the same as for
:(1x) 1-drive, 1-channel, 1-card.
:
:Hence my impression I can't use the two channels on one card
:simultaneously. Is this really different for you? Do you have the
:promise BIOS prom installed?
:
:Thanks for the fix, BTW, does this fix the IDE timeouts I see when
:striping over 4 drives?
:
:Martin
:--
You guys are all doing something wrong. First, everyone is using
tiny dd's to test. Second, one SCSI channel is plenty sufficient
to drive huge transfer rates through a 4-drive ccd stripe as I will
demonstrate with our backup1 machine. Third, raw devices impose
serialization limits and throughput depends heavily on the number of
processes accessing the device. i.e. you can't pipeline to or from a raw
device. Test it through the filesystem and read and write a lot of
data to get around caches.
Here are some tests I did with BEST's backup1 machine. It has a 36 GB
4xDisk stripe for a tape buffer. PPro 200. 4 x 9G drive
(three Seagate ST19171W, 7200 RPM drives and one ST39173W 3600 rpm drive).
All drives are connected to *ONE* SCSI ultra-wide controller
(one 40 MBytes/sec SCSI bus). The tape units are connected to a second
controller which is unused for all the tests below.
CCD: 128 sector (64 KByte) interleave across four drives.
And this is FreeBSD-stable, too, without CAM.
Test #1: read raw device, 64K blocks, 256 MBytes total:
backup1:/# dd if=/dev/rccd0d of=/dev/null bs=64k count=4096
268435456 bytes transferred in 8.617552 secs (31149851 bytes/sec)
31 MBytes/sec transfer rate. Because the disk does its own
internal read-ahead.
Test #2: write raw device, 64K blocks, 64 MBytes total:
backup1:/# dd if=/dev/zero of=/dev/rccd0d bs=64k count=1024
67108864 bytes transferred in 12.152601 secs (5522181 bytes/sec)
5.5 MBytes/sec, because the OS/disk doesn't cache writes through the
raw device.
Test #3: write raw device, 64K blocks, 64 MBytes total, 4 dd's in parallel
each doing 64 MBytes.
foreach? dd if=/dev/zero of=/dev/rccd0d bs=64k count=1024 seek=$i &
foreach? end
[1] 8465
[2] 8466
[3] 8467
[4] 8468
67108864 bytes transferred in 22.503316 secs (2982177 bytes/sec)
67108864 bytes transferred in 22.510251 secs (2981258 bytes/sec)
67108864 bytes transferred in 22.529739 secs (2978679 bytes/sec)
67108864 bytes transferred in 22.542779 secs (2976956 bytes/sec)
12 MBytes/sec, again because the OS/disk doesn't cache writes through
the raw device.
Test #4: filesystem throughput. 1 GByte total, write 64K blocks
newfs -i 32768 /dev/rccd0d
backup1:/ccd# dd if=/dev/zero of=/ccd/test1 bs=64k count=16384
1073741824 bytes transferred in 41.744593 secs (25721698 bytes/sec)
25 MBytes/sec writing. <--------------- write FS throughput
No chance of the OS or drive getting cache hits here, I'm writing a
friggin gigabyte file!
Test #5: filesystem throughput. 1 GByte total, read 64K blocks
backup1:/ccd# dd if=/ccd/test1 of=/dev/null bs=64k count=16384
1073741824 bytes transferred in 34.051957 secs (31532456 bytes/sec)
31 MBytes/sec reading. <-------------- read FS throughput
No chance of the OS or drive getting cache hits here, I'm reading
a friggin gigabyte file!
That's with *ONE* SCSI controller. An adaptec 2940UW.
This is one of several reasons why you use SCSI if you need performance.
You can't beat it for linear performance, and you can beat it for
transactional performance either. If you have just one drive then
sure, IDE will have similar linear performance. If you have two drives
you may even see similar linear performance reading, but I would guess
that you wouldn't see it writing. If you have several drives,
SCSI beats the shit out of IDE reading and writing.
For non-linear performance under load, the one-drive case will again
yield similar results SCSI vs IDE. But the moment you start trying
to random read two IDE devices on a single IDE controller, SCSI instantly
scales to the two drives while the two IDE devices will not perform much
better then one. After that, IDE goes into the shitpile for
non-linear performance tests due to its command serialization. A SCSI
bus with 4 drives on it will scale 4x for non-linear performance. Two
IDE busses with 4 drives total will only scale 2x for non-linear
performance. A SCSI bus with 8 drives on it will scale 8x for non-linear
performance.
I need non-linear performance on my web servers. My tape backup
machine needs linear performance on the disk buffer since it is throwing
multi-gigabyte files all over the place. A mail partition typically needs
linear performance as well... gotta be able to copy those 20 MByte
pop boxes :-).
-Matt
Matthew Dillon Engineering, HiWay Technologies, Inc. & BEST Internet
Communications & God knows what else.
<dillon@backplane.com> (Please include original email in any response)
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811111932.LAA22898>