Date: Tue, 15 Jan 2013 16:06:17 -0800 From: Dieter BSD <dieterbsd@gmail.com> To: freebsd-hackers@freebsd.org Subject: Re: IBM blade server abysmal disk write performances Message-ID: <CAA3ZYrD4EKrt9zbtp4UEbAhCCzvoMu4%2B6XBA515wFmeN--BH=w@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Karim writes:
> dd to the
> raw drive and no compression/encryption or some other features, just a
> naive boot off a live 9.1 CD then dd (see below). The following results
> have been gathered on the FreeBSD 9.1 system:
>
> # dd if=/dev/zero of=toto count=100
> 100+0 records in
> 100+0 records out
> 51200 bytes transferred in 1.057507 secs (48416 bytes/sec)
By "raw drive" I meant something like
dd if=/dev/zero of=/dev/da0 bs=1m count=1000
"of=toto" implies that you are using a filesystem. (FFS? ZFS? other?)
Matthew writes:
>> But dmesg claims "Command Queueing enabled", so you should be
>> getting more than one op per rev, and writes should be fast.
>
> Queueing would only help if your load threw multiple ops at the drive
> before waiting for any of them to complete. I'd expect a dd to a raw
> device to throw a single, wait for it to return complete, then throw
> the next, leading to no more than 1 op per rev.
I see a huge speedup from NCQ on both raw disks and with FFS/su.
Without NCQ I only get 6% of the expected performance, even with
a large blocksize.
The kernel must be doing write-behind even to a raw disk, otherwise
waiting for write(2) to return before issuing the next write would
slow it down as Matthew suggests.
Writing an entire 3TB disk (raw disk, no fs) gives:
21378.98 real 2.00 user 440.98 sys
or 140 MB/s (133 MiB/s) on slow controller in PCIe x1 slot.
The same test on the same make & model disk on a much faster controller
in the chipset takes over 10x as long because FreeBSD does not support
NCQ on that controller. :-(
Karim's data sure looks like 1 op per rev. Either it isn't really doing
NCQ or the filesystem is doing something to keep NCQ from being effective.
For example, mounting the fs with the sync option would probably have
that effect.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAA3ZYrD4EKrt9zbtp4UEbAhCCzvoMu4%2B6XBA515wFmeN--BH=w>