Date: Mon, 20 Sep 1999 15:24:37 +0930 From: Greg Lehey <grog@lemis.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Poul-Henning Kamp <phk@critter.freebsd.dk>, dg@root.com, cvs-committers@FreeBSD.ORG, cvs-all@FreeBSD.ORG Subject: Re: User block device access (was: cvs commit: src/sys/miscfs/specfs spec_vnops.c src/sys/sys vnode.h src/sys/kern vfs_subr.c) Message-ID: <19990920152437.W55065@freebie.lemis.com> In-Reply-To: <199909200527.WAA77336@apollo.backplane.com>; from Matthew Dillon on Sun, Sep 19, 1999 at 10:27:04PM -0700 References: <16748.937762251@critter.freebsd.dk> <199909191806.LAA73255@apollo.backplane.com> <19990920115556.J55065@freebie.lemis.com> <199909200527.WAA77336@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sunday, 19 September 1999 at 22:27:04 -0700, Matthew Dillon wrote:
>>
>> Poul-Henning has already replied to this with some figures. I can
>> confirm what he says; I've been looking a lot at this sort of thing
>> with Vinum. That doesn't answer the question "why", however. I
>> suspect that there's something funny in buffer management which is
>> reducing performance below what it could be. Vinum has a tool which
>> can be used to measure device queue length, and I've seen widely
>> differing queue lengths when writing to Vinum block devices: they can
>> fluctuate between 100 requests and almost none. I can't explain that
>> by the way Vinum works; it will just malloc and issue requests. About
>> the only place it could block is in malloc, but the actual memory
>> usage fluctuates wildly.
>>
>> I suspect that this is a bug rather than a feature; the question is,
>> is it worth following?
>
> It is, plain and simply, the fact that you are doing one copy (which
> is in fact direct DMA) in the case of raw I/O and doing two copies
> (one into the cache - direct DMA, and then a copy to the user buffer )
> in the case of buffered I/O.
That has nothing to do with the effect I noted, which may be a red
herring.
I've just checked, one reason for the slower transfer is because the
dd transfer from the raw device always reads 2 kB per transfer, no
matter what you tell dd. Here's iostat output for the following
commands:
dd if=/dev/wd3c of=/dev/null bs=64b
tty da0 da1 wd0 wd2 wd3 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
8 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.05 1290 2.59 2 0 18 1 79
4 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 1588 3.10 2 0 20 4 74
16 110 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 1602 3.13 2 0 25 3 70
dd if=/dev/rwd3c of=/dev/null bs=64b
0 108 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 312 9.74 2 0 5 1 92
148 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 311 9.71 3 0 4 0 93
371 108 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 312 9.74 5 0 11 1 83
I'm not sure why this is, but I suppose I could go look some time.
> If you apply the mmap() patches found on
> http://www.backplane.com/FreeBSD4/ and do an mmap based test, you will
> find that the overhead narrows considerably.
Hmmm. I tried running with 2kB blocks, and the results weren't what I
expected. I need to scratch my head some more. Here's what I got:
dd if=/dev/rwd3c of=/dev/null bs=4b count=10000
1 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 3891 7.60 2 0 28 4 67
0 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 3889 7.60 4 0 24 7 66
dd if=/dev/wd3c of=/dev/null bs=4b count=10000
193 325 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.10 143 0.29 4 0 2 1 93
2113 342 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.76 162 0.44 5 0 4 1 90
> There is nothing sinister going on here, buffer copying eats cpu
> plain and simple.
The primary effect here isn't buffer copying.
Greg
--
See complete headers for address, home page and phone numbers
finger grog@lemis.com for PGP public key
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe cvs-all" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19990920152437.W55065>