From owner-freebsd-fs@FreeBSD.ORG Sun Mar 6 18:48:19 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF9351065672 for ; Sun, 6 Mar 2011 18:48:19 +0000 (UTC) (envelope-from boydjd@jbip.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4A2A08FC1B for ; Sun, 6 Mar 2011 18:48:18 +0000 (UTC) Received: by fxm19 with SMTP id 19so3969675fxm.13 for ; Sun, 06 Mar 2011 10:48:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jbip.net; s=google; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=+AcuZp0Kb35R/Ocu/vMmOgXnRQsWV8yvHk8ozJxqFyI=; b=HXoF42t8my9jFhqoejCZryoKeJ1hgtdVzWmgLav2Pp1OZ/UrPXIVZ+t1VZw+cT1C83 RzI47ykTKKYTJAgB3TfY3tmMPH3PJsom/+HSoP2jjHozQLlEcIxTddxvxth6gsxVgT8D QcC5v8oUMqluwdWLHceMLtu1Xfq0OIpwWx9gM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=jbip.net; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=WdYMshgo35cd5avjolB+nAo4p7F06iuxPmEzBdGFdth9SS+0UMQeOkODKFgo1SROUi X3pXn+KQjJVr+w97HSmfetKhrh1fJT21el62RE46EitWIsjlyQ97gqtlATcEj7zQtq41 L68IIxpGSHiciOfTDqnRaO+PrR1YZmQK2mPXE= Received: by 10.223.151.14 with SMTP id a14mr1527634faw.134.1299437298067; Sun, 06 Mar 2011 10:48:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.223.144.137 with HTTP; Sun, 6 Mar 2011 10:47:58 -0800 (PST) In-Reply-To: <20110306090455.GA87055@icarus.home.lan> References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <20110304105608.GA23887@icarus.home.lan> <20110306090455.GA87055@icarus.home.lan> From: Joshua Boyd Date: Sun, 6 Mar 2011 13:47:58 -0500 Message-ID: To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 18:48:19 -0000 On Sun, Mar 6, 2011 at 4:04 AM, Jeremy Chadwick wrote: > On Sun, Mar 06, 2011 at 02:59:43AM -0500, Joshua Boyd wrote: >> On Fri, Mar 4, 2011 at 5:56 AM, Jeremy Chadwick >> wrote: >> > If you get better performance -- really, truly, honestly -- with >> > prefetch enabled on your system, then I strongly recommend you keep it >> > enabled. ?However, for what it's worth (probably not much), this is th= e >> > first I've ever heard of a FreeBSD system performing better with >> > prefetch enabled. >> >> I just recently turned it on after having it turned off for a long >> time ... my speeds went from ~300MB/s to 600+MB/s in bonnie++. This is >> a dual core AM3 system with 8GB of ram, and 15 disks in a striped >> raidz configuration (3 sets striped). > > Here are some numbers for you. =A0This is from a 8.2-STABLE (RELENG_8) > system built Thu Feb 24 22:06:45 PST 2011, type amd64. Interesting results. My kernel currently has a build date 2 days earlier than yours. Here are my results, showing the huge increase in speed. The only major configuration difference appears that I've disabled the ZIL and you have yours enabled. That shouldn't make any difference for read speeds though. FreeBSD foghornleghorn.res.openband.net 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #13: Tue Feb 22 17:39:03 EST 2011 root@foghornleghorn.res.openband.net:/usr/obj/usr/src/sys/FOGHORNLEGHORN amd64 /boot/loader.conf =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D vfs.zfs.zil_disable=3D"1" vfs.zfs.vdev.min_pending=3D"1" vfs.zfs.vdev.max_pending=3D"1" vm.kmem_size=3D"8192M" vfs.zfs.arc_max=3D6144M vfs.zfs.prefetch_disable=3D"0" vfs.zfs.txg.timeout=3D"5" /etc/sysctl.conf =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D kern.maxfiles=3D65536 kern.maxfilesperproc=3D32768 vfs.read_max=3D32 vfs.ufs.dirhash_maxmem=3D16777216 kern.maxvnodes=3D250000 vfs.zfs.txg.write_limit_override=3D1073741824 ZFS details =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 18.2T 7.73T 10.4T 42% ONLINE - # zfs list NAME USED AVAIL REFER MOUNTPOINT tank 6.17T 8.10T 36.7K /tank tank/downloads 4.89T 8.10T 2.30T /tank/downloads tank/downloads/movies 2.59T 8.10T 2.59T /tank/downloads/movies tank/usr 1.29T 8.10T 32.0K /tank/usr tank/usr/home 1.29T 8.10T 69.5K /usr/home tank/usr/home/josh 1.29T 8.10T 13.4G /usr/home/josh tank/usr/home/josh/hellanzb 32.0K 8.10T 32.0K /usr/home/josh/hellanzb tank/usr/home/josh/rtorrent 1.27T 8.10T 1.27T /usr/home/josh/rtorrent tank/usr/home/josh/watch 8.00M 8.10T 8.00M /usr/home/josh/watch # zpool status tank pool: tank state: ONLINE scrub: scrub completed after 7h43m with 0 errors on Sun Mar 6 07:43:56 20= 11 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da8 ONLINE 0 0 0 da18 ONLINE 0 0 0 da19 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da11 ONLINE 0 0 0 da10 ONLINE 0 0 0 da17 ONLINE 0 0 0 da9 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da4 ONLINE 0 0 0 errors: No known data errors Controller details =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D mpt0: port 0x6000-0x60ff mem 0xf75fc000-0xf75fffff,0xf75e0000-0xf75effff irq 18 at device 0.0 on pci1 mpt0: [ITHREAD] mpt0: MPI Version=3D1.5.20.0 mpt1: port 0x7000-0x70ff mem 0xf78fc000-0xf78fffff,0xf78e0000-0xf78effff irq 19 at device 0.0 on pci2 mpt1: [ITHREAD] mpt1: MPI Version=3D1.5.20.0 mpt2: port 0xd000-0xd0ff mem 0xf7ffc000-0xf7ffffff,0xf7fe0000-0xf7feffff irq 19 at device 0.0 on pci6 mpt2: [ITHREAD] mpt2: MPI Version=3D1.5.19.0 Disk details =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D da8 at mpt0 bus 0 scbus0 target 0 lun 0 da8: Fixed Direct Access SCSI-5 device da8: 300.000MB/s transfers da8: Command Queueing enabled da8: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da9 at mpt0 bus 0 scbus0 target 1 lun 0 da9: Fixed Direct Access SCSI-5 device da9: 300.000MB/s transfers da9: Command Queueing enabled da9: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da0 at mpt1 bus 0 scbus1 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da10 at mpt0 bus 0 scbus0 target 2 lun 0 da10: Fixed Direct Access SCSI-5 device da10: 300.000MB/s transfers da10: Command Queueing enabled da10: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da11 at mpt0 bus 0 scbus0 target 3 lun 0 da11: Fixed Direct Access SCSI-5 device da11: 300.000MB/s transfers da11: Command Queueing enabled da11: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da1 at mpt1 bus 0 scbus1 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing enabled da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da2 at mpt1 bus 0 scbus1 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 300.000MB/s transfers da2: Command Queueing enabled da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da3 at mpt1 bus 0 scbus1 target 3 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 300.000MB/s transfers da3: Command Queueing enabled da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da4 at mpt1 bus 0 scbus1 target 4 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 300.000MB/s transfers da4: Command Queueing enabled da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da5 at mpt1 bus 0 scbus1 target 5 lun 0 da5: Fixed Direct Access SCSI-5 device da5: 300.000MB/s transfers da5: Command Queueing enabled da5: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da6 at mpt1 bus 0 scbus1 target 6 lun 0 da6: Fixed Direct Access SCSI-5 device da6: 300.000MB/s transfers da6: Command Queueing enabled da6: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da7 at mpt1 bus 0 scbus1 target 7 lun 0 da7: Fixed Direct Access SCSI-5 device da7: 300.000MB/s transfers da7: Command Queueing enabled da7: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da16 at mpt2 bus 0 scbus2 target 82 lun 0 da16: Fixed Direct Access SCSI-5 device da16: 300.000MB/s transfers da16: Command Queueing enabled da16: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da17 at mpt2 bus 0 scbus2 target 83 lun 0 da17: Fixed Direct Access SCSI-5 device da17: 300.000MB/s transfers da17: Command Queueing enabled da17: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da18 at mpt2 bus 0 scbus2 target 84 lun 0 da18: Fixed Direct Access SCSI-5 device da18: 300.000MB/s transfers da18: Command Queueing enabled da18: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) da19 at mpt2 bus 0 scbus2 target 85 lun 0 da19: Fixed Direct Access SCSI-5 device da19: 300.000MB/s transfers da19: Command Queueing enabled da19: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) Benchmark results #1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Version 1.96 ------Sequential Output------ --Sequential Input- --Ran= dom- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --See= ks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec= %CP foghornleghorn. 16G 213 99 266782 53 90296 19 480 95 218719 24 229.7 6 Latency 43348us 37929us 242ms 102ms 68306us 462= ms Version 1.96 ------Sequential Create------ --------Random Create----= ---- foghornleghorn.res. -Create-- --Read--- -Delete-- -Create-- --Read--- -Dele= te-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec= %CP 16 15220 44 +++++ +++ 19214 56 22371 58 +++++ +++ 22133= 66 Latency 10658us 60us 82us 6540us 39us 1677= us Benchmark results #2 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Version 1.96 ------Sequential Output------ --Sequential Input- --Ran= dom- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --See= ks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec= %CP foghornleghorn. 16G 201 99 276506 56 198428 38 459 97 627451 73 252.0 5 Latency 45695us 35953us 265ms 69630us 42440us 389= ms Version 1.96 ------Sequential Create------ --------Random Create----= ---- foghornleghorn.res. -Create-- --Read--- -Delete-- -Create-- --Read--- -Dele= te-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec= %CP 16 14988 50 +++++ +++ 18693 59 18535 51 +++++ +++ 20827= 67 Latency 13309us 93us 116us 8165us 36us 1046= us --=20 Joshua Boyd JBipNet E-mail: boydjd@jbip.net http://www.jbip.net