From owner-freebsd-fs@FreeBSD.ORG  Thu Nov  5 06:25:38 2009
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 38744106566C
	for <freebsd-fs@freebsd.org>; Thu,  5 Nov 2009 06:25:38 +0000 (UTC)
	(envelope-from ai@kliksys.ru)
Received: from gate.kliksys.ru (gate.kliksys.ru [78.110.241.113])
	by mx1.freebsd.org (Postfix) with ESMTP id E0DBA8FC17
	for <freebsd-fs@freebsd.org>; Thu,  5 Nov 2009 06:25:37 +0000 (UTC)
Received: from [192.168.1.13] (helo=one.kliksys.ru)
	by gate.kliksys.ru with esmtp (Exim 4.69 (FreeBSD))
	(envelope-from <ai@kliksys.ru>) id 1N5vc6-000BwW-Qg
	for freebsd-fs@freebsd.org; Thu, 05 Nov 2009 09:14:31 +0300
Date: Thu, 5 Nov 2009 09:14:30 +0300
From: Artemiev Igor <ai@kliksys.ru>
To: freebsd-fs@freebsd.org
Message-ID: <20091105061430.GA92808@one.kliksys.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.19 (2009-01-05)
X-Spam_score: 0.0
Subject: Low zfs prefetch hits - why?
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 05 Nov 2009 06:25:38 -0000


#sysctl vfs.zfs.zfetch
vfs.zfs.zfetch.array_rd_sz: 524288
vfs.zfs.zfetch.block_cap: 8
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8

Trying to sequential read 1G file by 128K chunks.
#./arcstat.pl -f Time,pmis,pm%
	Time  pmis  pm%
    05:48:40   749  100
    05:48:41    1K  100
    05:48:42    1K  100
    05:48:43    1K  100
    05:48:44    1K  100
    05:48:45    1K  100
    05:48:46    1K  100
    05:48:47   257   99

I thought, blocks didn`t prefetching and wrote small dtrace script:

#!/usr/sbin/dtrace -qs

fbt:zfs:dmu_zfetch:entry
{
    printf("dmu_zfetch: offset=%ld size=%ld prefetched=%ld\n", arg1, arg2, arg3);
}


fbt:zfs:dbuf_prefetch:entry
{
    printf("  zfetching block %d\n", arg1);
}

Here output:
dmu_zfetch: offset=325058560 size=131072 prefetched=32
dmu_zfetch: offset=325189632 size=131072 prefetched=32
dmu_zfetch: offset=325320704 size=131072 prefetched=32
dmu_zfetch: offset=325451776 size=131072 prefetched=32
dmu_zfetch: offset=325582848 size=131072 prefetched=32
dmu_zfetch: offset=326369280 size=131072 prefetched=32
dmu_zfetch: offset=326500352 size=131072 prefetched=32
dmu_zfetch: offset=326631424 size=131072 prefetched=32
dmu_zfetch: offset=326762496 size=131072 prefetched=32
dmu_zfetch: offset=326893568 size=131072 prefetched=32
dmu_zfetch: offset=327024640 size=131072 prefetched=32
  zfetching block 2503
  zfetching block 2504
  zfetching block 2505
  zfetching block 2506
  zfetching block 2507
  zfetching block 2508
  zfetching block 2509
  zfetching block 2510
dmu_zfetch: offset=327155712 size=131072 prefetched=32
dmu_zfetch: offset=327286784 size=131072 prefetched=32
dmu_zfetch: offset=327417856 size=131072 prefetched=32
dmu_zfetch: offset=327548928 size=131072 prefetched=32
dmu_zfetch: offset=327680000 size=131072 prefetched=32

Why this happening? Statistic is unrelevant?
With sendfile(2) prefetching completely didn`t work - no forward read ahead.