Date: Thu, 30 Aug 2007 08:40:18 -0500 From: Eric Anderson <anderson@freebsd.org> To: Scott Long <scottl@samsco.org> Cc: FREEBSD - SCSI - LIST <freebsd-scsi@freebsd.org> Subject: Re: performance with LSI SAS 1064 Message-ID: <46D6C8C2.3000504@freebsd.org> In-Reply-To: <46D64020.3000503@samsco.org> References: <71d0ebb0708291245g79d2141fx73cc8a6e76875944@mail.gmail.com> <46D5E17F.3070403@samsco.org> <71d0ebb0708291416v17351c65u7ccc1b7bbe0271d2@mail.gmail.com> <46D5E5B1.207@samsco.org> <71d0ebb0708291506i49649a60l8006deafb20891ac@mail.gmail.com> <46D63710.1020103@freebsd.org> <46D64020.3000503@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Scott Long wrote:
> Eric Anderson wrote:
>> Lutieri G. wrote:
>>> I've make a test with dd command:
>>>
>>> # time dd if=/dev/zero of=./8gbfile bs=1024k count=8192
>>> 8192+0 records in
>>> 8192+0 records out
>>> 8589934592 bytes transferred in 155.653213 secs (55186362 bytes/sec)
>>> 0.007u 25.129s 2:35.69 16.1% 55+6039k 117+68628io 0pf+0w
>>>
>>> in other terminal i ran iostat while dd were running and I get this:
>>>
>>> # iostat -I 1
>>> tty da0 pass0 cpu
>>> tin tout KB/t xfrs MB KB/t xfrs MB us ni sy in id
>>> 1 61 25.23 52958 1305.07 0.00 0 0.00 0 0 8 0 91
>>> 0 184 127.48 434 54.03 0.00 0 0.00 0 0 5 0 95
>>> 0 61 127.49 440 54.78 0.00 0 0.00 0 0 5 0 95
>>> 0 61 127.75 445 55.52 0.00 0 0.00 0 0 5 0 95
>>> 0 61 127.49 442 55.03 0.00 0 0.00 0 0 4 0 96
>>> 0 61 127.49 436 54.28 0.00 0 0.00 0 0 5 0 95
>>> 0 61 125.27 425 51.99 0.00 0 0.00 0 0 5 1 94
>>> 0 61 118.14 393 45.34 0.00 0 0.00 0 0 3 0 97
>>>
>>> average 54MB/s with or without hw.mpt.enable_sata_wc seted in
>>> loader.conf file.
>>>
>>> is it a normal speed for this adapter?!
>>>
>>>
>>
>>
>> I'm confused - you said in your first post you were getting 3MB/s,
>> where above you show something like 55MB/s.
>>
>> You didn't say what kind of disks, or how many, the configuration, etc
>> - so it's hard to answer much. The 55MB/s seems pretty decent for
>> many hard drives in a sequential use state (which is what dd tests
>> really).
>>
>> Your errors before were probably caused because your queue depth is
>> set to 255 (or 256?) and the adapter can't do that many. You should
>> use camcontrol to reduce it, to maybe 32. See the camcontrol man page
>> for the right usage. It's something that needs setting on every boot,
>> so a startup file is a good place for it maybe.
>>
>> Eric
>>
>>
>
> Well, if he's using SATA (which I kinda assumed originally without
> asking) then queue depth isn't going to matter; the MPT driver has no
> interaction with how SATA NCQ operates, if he even has a rev of the
> LSI chip that supports NCQ at all. If he's using SAS, then queue
> depth will only be a minor factor, CAM is pretty good at autosizing
> the depth with minimal impact. Now, if he's using SAS disks then
> the boot tunable that I gave him will indeed have no impact at all.
>
> I believe that the Sun 4100 uses 2.5" disks, whether SATA or SAS.
> 54MB/s is not all that bad for disks of this size. It's pretty close
> to what I would expect, actually.
If he's using a SAS Seagate 15k rpm 2.5" drive, he could see much better
than 55MB/s. I have an LSI (PCI-X 133, model 1064 as he does) with some
Seagate 15k RPM drives, and I can get 100MB/s. Tests were done on
FreeBSD 7-CURRENT, with write caching enabled on the drive. Here's some
of the numbers:
1k block read size: 6390033 bytes/sec
2k block read size: 12257594 bytes/sec
4k block read size: 23071775 bytes/sec
8k block read size: 41858586 bytes/sec
16k block read size: 71242623 bytes/sec
32k block read size: 108019212 bytes/sec
64k block read size: 108043811 bytes/sec
128k block read size: 108881295 bytes/sec
256k block read size: 108509827 bytes/sec
512k block read size: 108670186 bytes/sec
1024k block read size: 108553724 bytes/sec
1k block write size: 6199645 bytes/sec
2k block write size: 12169083 bytes/sec
4k block write size: 23246388 bytes/sec
8k block write size: 42133751 bytes/sec
16k block write size: 70968273 bytes/sec
32k block write size: 108306742 bytes/sec
64k block write size: 108142985 bytes/sec
128k block write size: 109156720 bytes/sec
256k block write size: 109164252 bytes/sec
512k block write size: 109795665 bytes/sec
1024k block write size: 109202660 bytes/sec
Doing latency tests on /dev/da0
/dev/da0
512 # sectorsize
73407820800 # mediasize in bytes (68G)
143374650 # mediasize in sectors
8924 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
Seek times:
Full stroke: 250 iter in 1.943102 sec = 7.772 msec
Half stroke: 250 iter in 1.567395 sec = 6.270 msec
Quarter stroke: 500 iter in 1.671074 sec = 3.342 msec
Short forward: 400 iter in 1.159763 sec = 2.899 msec
Short backward: 400 iter in 1.014000 sec = 2.535 msec
Seq outer: 2048 iter in 0.322013 sec = 0.157 msec
Seq inner: 2048 iter in 0.493810 sec = 0.241 msec
Transfer rates:
outside: 102400 kbytes in 0.968299 sec = 105752 kbytes/sec
middle: 102400 kbytes in 1.088158 sec = 94104 kbytes/sec
inside: 102400 kbytes in 1.361127 sec = 75232 kbytes/sec
Doing read tests on /dev/da0
READ: 1k block size:
102400+0 records in
102400+0 records out
104857600 bytes transferred in 16.409555 secs (6390033 bytes/sec)
READ: 2k block size:
51200+0 records in
51200+0 records out
104857600 bytes transferred in 8.554501 secs (12257594 bytes/sec)
READ: 4k block size:
25600+0 records in
25600+0 records out
104857600 bytes transferred in 4.544843 secs (23071775 bytes/sec)
READ: 8k block size:
12800+0 records in
12800+0 records out
104857600 bytes transferred in 2.505044 secs (41858586 bytes/sec)
READ: 16k block size:
6400+0 records in
6400+0 records out
104857600 bytes transferred in 1.471838 secs (71242623 bytes/sec)
READ: 32k block size:
3200+0 records in
3200+0 records out
104857600 bytes transferred in 0.970731 secs (108019212 bytes/sec)
READ: 64k block size:
1600+0 records in
1600+0 records out
104857600 bytes transferred in 0.970510 secs (108043811 bytes/sec)
READ: 128k block size:
800+0 records in
800+0 records out
104857600 bytes transferred in 0.963045 secs (108881295 bytes/sec)
READ: 256k block size:
400+0 records in
400+0 records out
104857600 bytes transferred in 0.966342 secs (108509827 bytes/sec)
READ: 512k block size:
200+0 records in
200+0 records out
104857600 bytes transferred in 0.964916 secs (108670186 bytes/sec)
READ: 1024k block size:
100+0 records in
100+0 records out
104857600 bytes transferred in 0.965951 secs (108553724 bytes/sec)
Doing write tests on /dev/da0
WRITE: 1k block size:
102400+0 records in
102400+0 records out
104857600 bytes transferred in 16.913484 secs (6199645 bytes/sec)
WRITE: 2k block size:
51200+0 records in
51200+0 records out
104857600 bytes transferred in 8.616722 secs (12169083 bytes/sec)
WRITE: 4k block size:
25600+0 records in
25600+0 records out
104857600 bytes transferred in 4.510705 secs (23246388 bytes/sec)
WRITE: 8k block size:
12800+0 records in
12800+0 records out
104857600 bytes transferred in 2.488684 secs (42133751 bytes/sec)
WRITE: 16k block size:
6400+0 records in
6400+0 records out
104857600 bytes transferred in 1.477528 secs (70968273 bytes/sec)
WRITE: 32k block size:
3200+0 records in
3200+0 records out
104857600 bytes transferred in 0.968154 secs (108306742 bytes/sec)
WRITE: 64k block size:
1600+0 records in
1600+0 records out
104857600 bytes transferred in 0.969620 secs (108142985 bytes/sec)
WRITE: 128k block size:
800+0 records in
800+0 records out
104857600 bytes transferred in 0.960615 secs (109156720 bytes/sec)
WRITE: 256k block size:
400+0 records in
400+0 records out
104857600 bytes transferred in 0.960549 secs (109164252 bytes/sec)
WRITE: 512k block size:
200+0 records in
200+0 records out
104857600 bytes transferred in 0.955025 secs (109795665 bytes/sec)
WRITE: 1024k block size:
100+0 records in
100+0 records out
104857600 bytes transferred in 0.960211 secs (109202660 bytes/sec)
Tests complete.
Eric
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46D6C8C2.3000504>