From owner-freebsd-current@FreeBSD.ORG Wed Apr 23 14:17:37 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EEC9106564A for ; Wed, 23 Apr 2008 14:17:37 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id 10E928FC13 for ; Wed, 23 Apr 2008 14:17:36 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper [152.3.145.30]) by duke.cs.duke.edu (8.14.2/8.14.2) with ESMTP id m3NEHakr017192 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 23 Apr 2008 10:17:36 -0400 (EDT) X-DKIM: Sendmail DKIM Filter v2.5.3 duke.cs.duke.edu m3NEHakr017192 Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id m3NEH8qk015042; Wed, 23 Apr 2008 10:17:08 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18447.17658.759349.720175@grasshopper.cs.duke.edu> Date: Wed, 23 Apr 2008 10:17:07 -0400 (EDT) To: freebsd-current@freebsd.org X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: ZFS file caching question X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Apr 2008 14:17:37 -0000 If I have, say, 512MB RAM and a 1GB file which is written or read sequentially on an otherwise idle system, I'd expect the last 512MB (- epsilon) of the file's pages to be cached in RAM. This is true for UFS, but it does not appear to be the case with ZFS (see example below). Can somebody explain how the arc cache in ZFS relates to the normal page cache used by traditional filesystems? Are ZFS filesystems cached exclusively in the arc cache, and not in the page cache? Is the arc cache per-filesystem, per-pool, or global for ZFS as a whole? Hmm.. Could this be the cause of the problems with ZFS and mmap'ed files? Thanks, Drew ### host has 406MB free before file is written %vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 4 74232K 406M 309 0 1 11 1747 2518 0 0 358 303 3267 0 3 97 %dd if=/dev/zero of=zot bs=1m count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 21.076331 secs (50945386 bytes/sec) ### host has 26MB free after file is written %vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 6 74232K 26M 303 0 1 11 1707 2565 0 0 359 299 3252 0 3 97 ### at least the last 300MB or so are cached, and the disk is not hit ### when reading them. %dd if=zot of=/dev/null iseek=700 bs=1m 324+0 records in 324+0 records out 339738624 bytes transferred in 0.372665 secs (911646356 bytes/sec) % But with ZFS, this is not true: ### host has 412MB free before file is written %vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 6 84552K 412M 263 0 1 9 1520 2200 0 0 309 263 2854 0 3 97 %dd if=/dev/zero of=zot bs=1m count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 17.447763 secs (61540372 bytes/sec) % ### host has 254MB free after file is written %vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 6 84552K 254M 249 0 1 9 1441 2069 0 0 299 251 2776 0 3 97 %dd if=zot of=/dev/null iseek=700 bs=1m 324+0 records in 324+0 records out 339738624 bytes transferred in 4.545527 secs (74741306 bytes/sec) ### the last 300MB of the file are not cached, and the disk is ### hit when reading them %vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 6 84552K 254M 249 0 1 9 1441 2069 0 0 299 251 2776 0 3 97 %dd if=zot of=/dev/null iseek=700 bs=1m 324+0 records in 324+0 records out 339738624 bytes transferred in 4.545527 secs (74741306 bytes/sec)