From owner-freebsd-current@FreeBSD.ORG Sun Jun 20 18:35:19 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51D1616A4CF; Sun, 20 Jun 2004 18:35:19 +0000 (GMT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4404943D2F; Sun, 20 Jun 2004 18:35:12 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) i5KIZBx7026533; Sun, 20 Jun 2004 11:35:11 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id i5KIZBeJ026532; Sun, 20 Jun 2004 11:35:11 -0700 (PDT) (envelope-from dillon) Date: Sun, 20 Jun 2004 11:35:11 -0700 (PDT) From: Matthew Dillon Message-Id: <200406201835.i5KIZBeJ026532@apollo.backplane.com> To: Mikhail Teterin References: <200406200343.03920@aldan> cc: questions@freebsd.org cc: current@freebsd.org Subject: Re: read vs. mmap (or io vs. page faults) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jun 2004 18:35:19 -0000 :Hello! : :I'm writing a message-digest utility, which operates on file and :can use either stdio: : : while (not eof) { : char buffer[BUFSIZE]; : size = read(.... buffer ...); : process(buffer, size); : } : :or mmap: : : buffer = mmap(... file_size, PROT_READ ...); : process(buffer, file_size); : :I expected the second way to be faster, as it is supposed to avoid :one memory copying (no user-space buffer). But in reality, on a :CPU-bound (rather than IO-bound) machine, using mmap() is considerably :slower. Here are the tcsh's time results: read() is likely going to be faster because it does not involve any page fault overhead. The VM system only faults 16 or so pages ahead which is only 64KB, so the fault overhead is very high for the data rate. Why does the extra copy not matter? Well, it's fairly simple, actually. It's because your buffer is smaller then the L1 cache, and/or also simply because the VM fault overhead is higher then it would take to copy an extra 64KB. read() loops typically use buffer sizes in the 8K-46K range. L1 caches are typically 16K (for celeron class cpus) through 64K, or more for higher end cpus. L2 caches are typically 256K-1MB, or more. The copy bandwidth from or to the L1 cache is usually around 10x faster then main memory and the copy bandwidth from or two L2 cache is usually around 4x faster. (Note that I'm talking copy bandwidth here, not random access. The L1 cache is ~50x faster or more for random access). So the cost of the extra copy in a read() loop using a reasonable buffer size (~8K-64K) (L1 or L2 access) is virtually nil compared to the cost of accessing the kernel's buffer cache (which involves main memory accesses for files > L2 cache). :On the IO-bound machine, using mmap is only marginally faster: : : Single Pentium4M (Centrino 1GHz) runing recent -current: : -------------------------------------------------------- :stdio: 27.195u 8.280s 1:33.02 38.1% 10+169k 11221+0io 1pf+0w :mmap: 26.619u 3.004s 1:23.59 35.4% 10+169k 47+0io 19463pf+0w Yes, because it's I/O bound. As long as the kernel queues some readahead to the device it can burn those cpu cycles on whatever it wants without really effecting the transfer rate. :I this how things are supposed to be, or will mmap() become more :efficient eventually? Thanks! : : -mi It's hard to say. mmap() could certainly be made more efficient, e.g. by faulting in more pages at a time to reduce the actual fault rate. But it's fairly difficult to beat a read copy into a small buffer. -Matt Matthew Dillon