From owner-freebsd-fs@FreeBSD.ORG Mon Mar 7 18:41:41 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 584191065675 for ; Mon, 7 Mar 2011 18:41:41 +0000 (UTC) (envelope-from mgamsjager@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0E2EA8FC12 for ; Mon, 7 Mar 2011 18:41:40 +0000 (UTC) Received: by gxk7 with SMTP id 7so1967999gxk.13 for ; Mon, 07 Mar 2011 10:41:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=oG6eqDq6RXeRn1p12/2vy1cDa6/ajA9flvq5E6NuAHE=; b=dmA9vtwMvQ2xXIplHP8lFAaTEaKt7HJvD8Q0Vfgg3iyNkSbZ3eiMkYRy+rCTEdhTvk EypE4JTgoQtTND0grsADSdgsgOqShTazEpSOggZzxVf7GLw8uoEDEhhjFzHA1cufpjGe 72OI0uJKw36c3OPSDOxbnEXA+Iq+IFPw4GO0M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=NfcMJWieZZaPZZOU7HEGneXaIgc2j6Mcd1W2rbZnIpmwfseORulvO166LhOY2vQleo j/h2KB1EuaIBaHDiB2lLGuYTEWYbIrRv4r1/WWvEMOYOtLD3cQYg6nmkroif+9/PcdxG FOHKR2+HE9yWk3stN7J7DsNvq/1+1RR7FDWng= Received: by 10.100.36.8 with SMTP id j8mr1507275anj.54.1299523300095; Mon, 07 Mar 2011 10:41:40 -0800 (PST) MIME-Version: 1.0 Received: by 10.100.205.10 with HTTP; Mon, 7 Mar 2011 10:41:10 -0800 (PST) In-Reply-To: References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <20110304105608.GA23887@icarus.home.lan> <20110306090455.GA87055@icarus.home.lan> From: Matthias Gamsjager Date: Mon, 7 Mar 2011 19:41:10 +0100 Message-ID: To: Joshua Boyd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 18:41:41 -0000 Let me too backup my claim with data: AMD Dual core 4G ram 4x 1TB Samsung drives OS installed on separate ufs disk FreeBSD fb 8.2-STABLE FreeBSD 8.2-STABLE #0 r219265: Fri Mar 4 16:47:35 CET 2011 loader.conf: vm.kmem_size="6G" vfs.zfs.txg.timeout="5" vfs.zfs.vdev.min_pending=1 #default = 4 vfs.zfs.vdev.max_pending=4 #default= 35 sysctl.conf: vfs.zfs.txg.write_limit_override=805306368 kern.sched.preempt_thresh=220 Zpool: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad10 ONLINE 0 0 0 mirror ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad8 ONLINE 0 0 0 NAME SIZE USED AVAIL CAP HEALTH ALTROOT storage 1.81T 1.57T 245G 86% ONLINE - Prefetch disable = 1 Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP fb 10000M 54 74 99180 42 35955 14 140 73 68174 11 180.6 4 Latency 295ms 1581ms 1064ms 428ms 58640us 755ms Version 1.96 ------Sequential Create------ --------Random Create-------- fb -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6697 39 +++++ +++ 11798 74 10060 61 +++++ +++ 11104 72 Latency 213ms 134us 257us 32866us 2672us 174us Prefetch disable = 0 Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP fb 10000M 52 74 107602 46 65443 29 135 74 243760 42 186.5 4 Latency 214ms 865ms 1525ms 79771us 254ms 924ms Version 1.96 ------Sequential Create------ --------Random Create-------- fb -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8152 56 +++++ +++ 4534 36 10966 69 32607 74 9692 71 Latency 112ms 21108us 169ms 30018us 4097us 318us Read performance 68MB/s vs 243MB/s. Maybe the kind of workload you have does not work well with prefetch, I don't know, but for sequential load like I use my NAS for, which is used as a media tank, it does boost performance quiet a bit.