From owner-freebsd-fs@FreeBSD.ORG Tue Aug 26 22:20:36 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C306A1065686 for ; Tue, 26 Aug 2008 22:20:36 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (host-122-100-2-232.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id 6F8268FC14 for ; Tue, 26 Aug 2008 22:20:36 +0000 (UTC) (envelope-from andrew@modulus.org) Received: by email.octopus.com.au (Postfix, from userid 1002) id 2FE4E172FC; Wed, 27 Aug 2008 08:02:00 +1000 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.1.50.60] (138.21.96.58.exetel.com.au [58.96.21.138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id DBF461712C for ; Wed, 27 Aug 2008 08:01:55 +1000 (EST) Message-ID: <48B47D42.9060307@modulus.org> Date: Wed, 27 Aug 2008 08:01:38 +1000 From: Andrew Snow User-Agent: Thunderbird 2.0.0.14 (X11/20080523) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Poor ZFS prefetch performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Aug 2008 22:20:36 -0000 On latest 8-current with latest patches, I am seeing poor data prefetch hit rate with my workload of rsync on millions of files. ZFS has been quite good apart from this - UFS was much faster at this particular workload. ZFS just seems to be reading many megabytes over and over from the filesystem with little to show for it. Arcstats reports the following: kstat.zfs.misc.arcstats.prefetch_data_hits: 475476 kstat.zfs.misc.arcstats.prefetch_data_misses: 5325057 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 126411515 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 3998476 I don't really want to turn of prefetching completely because it does actually help in many situations. From this I deduce it would be really nice to provide an extra tunable that allows you to turn off prefetching for data but leave it on for metadata. This would be easy to do because data prefetches come in through dmu_zfetch, but metadata comes through dmu_prefetch() in dmu.c - Andrew