From owner-cvs-all Sun Sep 19 22:55: 6 1999 Delivered-To: cvs-all@freebsd.org Received: from freebie.lemis.com (freebie.lemis.com [192.109.197.137]) by hub.freebsd.org (Postfix) with ESMTP id C899A1570A; Sun, 19 Sep 1999 22:54:58 -0700 (PDT) (envelope-from grog@freebie.lemis.com) Received: (from grog@localhost) by freebie.lemis.com (8.9.3/8.9.0) id PAA08708; Mon, 20 Sep 1999 15:24:37 +0930 (CST) Date: Mon, 20 Sep 1999 15:24:37 +0930 From: Greg Lehey To: Matthew Dillon Cc: Poul-Henning Kamp , dg@root.com, cvs-committers@FreeBSD.ORG, cvs-all@FreeBSD.ORG Subject: Re: User block device access (was: cvs commit: src/sys/miscfs/specfs spec_vnops.c src/sys/sys vnode.h src/sys/kern vfs_subr.c) Message-ID: <19990920152437.W55065@freebie.lemis.com> References: <16748.937762251@critter.freebsd.dk> <199909191806.LAA73255@apollo.backplane.com> <19990920115556.J55065@freebie.lemis.com> <199909200527.WAA77336@apollo.backplane.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95.4i In-Reply-To: <199909200527.WAA77336@apollo.backplane.com>; from Matthew Dillon on Sun, Sep 19, 1999 at 10:27:04PM -0700 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-41-739-7062 Sender: owner-cvs-all@FreeBSD.ORG Precedence: bulk On Sunday, 19 September 1999 at 22:27:04 -0700, Matthew Dillon wrote: >> >> Poul-Henning has already replied to this with some figures. I can >> confirm what he says; I've been looking a lot at this sort of thing >> with Vinum. That doesn't answer the question "why", however. I >> suspect that there's something funny in buffer management which is >> reducing performance below what it could be. Vinum has a tool which >> can be used to measure device queue length, and I've seen widely >> differing queue lengths when writing to Vinum block devices: they can >> fluctuate between 100 requests and almost none. I can't explain that >> by the way Vinum works; it will just malloc and issue requests. About >> the only place it could block is in malloc, but the actual memory >> usage fluctuates wildly. >> >> I suspect that this is a bug rather than a feature; the question is, >> is it worth following? > > It is, plain and simply, the fact that you are doing one copy (which > is in fact direct DMA) in the case of raw I/O and doing two copies > (one into the cache - direct DMA, and then a copy to the user buffer ) > in the case of buffered I/O. That has nothing to do with the effect I noted, which may be a red herring. I've just checked, one reason for the slower transfer is because the dd transfer from the raw device always reads 2 kB per transfer, no matter what you tell dd. Here's iostat output for the following commands: dd if=/dev/wd3c of=/dev/null bs=64b tty da0 da1 wd0 wd2 wd3 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 8 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.05 1290 2.59 2 0 18 1 79 4 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 1588 3.10 2 0 20 4 74 16 110 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 1602 3.13 2 0 25 3 70 dd if=/dev/rwd3c of=/dev/null bs=64b 0 108 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 312 9.74 2 0 5 1 92 148 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 311 9.71 3 0 4 0 93 371 108 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 32.00 312 9.74 5 0 11 1 83 I'm not sure why this is, but I suppose I could go look some time. > If you apply the mmap() patches found on > http://www.backplane.com/FreeBSD4/ and do an mmap based test, you will > find that the overhead narrows considerably. Hmmm. I tried running with 2kB blocks, and the results weren't what I expected. I need to scratch my head some more. Here's what I got: dd if=/dev/rwd3c of=/dev/null bs=4b count=10000 1 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 3891 7.60 2 0 28 4 67 0 109 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.00 3889 7.60 4 0 24 7 66 dd if=/dev/wd3c of=/dev/null bs=4b count=10000 193 325 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.10 143 0.29 4 0 2 1 93 2113 342 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2.76 162 0.44 5 0 4 1 90 > There is nothing sinister going on here, buffer copying eats cpu > plain and simple. The primary effect here isn't buffer copying. Greg -- See complete headers for address, home page and phone numbers finger grog@lemis.com for PGP public key To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe cvs-all" in the body of the message