From owner-freebsd-performance@FreeBSD.ORG Fri Mar 19 21:40:31 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34489106564A; Fri, 19 Mar 2010 21:40:31 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 808F18FC1C; Fri, 19 Mar 2010 21:40:30 +0000 (UTC) Received: by bwz8 with SMTP id 8so3446544bwz.3 for ; Fri, 19 Mar 2010 14:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=VtYu6IlkRcrTpjamNmL7ifAXdEWEVeuZkVCUKd06v1k=; b=ijz+NKqoXWkZutS6lPlKUEtDKCs4ck+1/p5/XZ3juAhvBSBt3mzFE36YdUJu71Jjlh gITWTtykZyZGq9AIrIZzltARRdx2fqjs/tIoJBG0GSpX0RQ+c2NRlh5J6SK1bwJtea2T hczGm5pIBSq3SvjURZ42gULHnxrReyHTx4SxM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=ANKV6hYVoEMX+Caef8LR7TNcEHYD23tUdMTQfSQq0MyTHKUuUIbZiV0eOYfavHyAlC 7bf19ZG3xyvjbu1H51oGFxuMMrca7n5pnVtd07HERmZ9BHFzoR5ZMYf1ajCTbrzsTCMr UswHies4ahCgS5DAr5uVKhL5ArGO8daWRV7NU= MIME-Version: 1.0 Received: by 10.204.85.73 with SMTP id n9mr1890804bkl.71.1269033288061; Fri, 19 Mar 2010 14:14:48 -0700 (PDT) Date: Fri, 19 Mar 2010 23:14:47 +0200 Message-ID: From: Dan Naumov To: freebsd-net@freebsd.org, freebsd-questions@freebsd.org, FreeBSD-STABLE Mailing List , freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Mailman-Approved-At: Fri, 19 Mar 2010 21:43:46 +0000 Cc: Subject: Samba read speed performance tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 21:40:31 -0000 On a FreeBSD 8.0-RELEASE/amd64 system with a Supermicro X7SPA-H board using an Intel gigabit nic with the em driver, running on top of a ZFS mirror, I was seeing a strange issue. Local reads and writes to the pool easily saturate the disks with roughly 75mb/s throughput, which is roughly the best these drives can do. However, working with Samba, writes to a share could easily pull off 75mb/s and saturate the disks, but reads off a share were resulting in rather pathetic 18mb/s throughput. I found a threadon the FreeBSD forums (http://forums.freebsd.org/showthread.php?t=9187) and followed the suggested advice. I rebuilt Samba with AIO support, kldloaded the aio module and made the following changes to my smb.conf From: socket options=TCP_NODELAY To: socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY min receivefile size=16384 use sendfile=true aio read size = 16384 aio write size = 16384 aio write behind = true dns proxy = no[/CODE] This showed a very welcome improvement in read speed, I went from 18mb/s to 48mb/s. The write speed remained unchanged and was still saturating the disks. Now I tried the suggested sysctl tunables: atombsd# sysctl net.inet.tcp.delayed_ack=0 net.inet.tcp.delayed_ack: 1 -> 0 atombsd# sysctl net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.path_mtu_discovery: 1 -> 0 atombsd# sysctl net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.recvbuf_inc: 16384 -> 524288 atombsd# sysctl net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvbuf_max: 262144 -> 16777216 atombsd# sysctl net.inet.tcp.sendbuf_inc=524288 net.inet.tcp.sendbuf_inc: 8192 -> 524288 atombsd# sysctl net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.sendbuf_max: 262144 -> 16777216 atombsd# sysctl net.inet.tcp.sendspace=65536 net.inet.tcp.sendspace: 32768 -> 65536 atombsd# sysctl net.inet.udp.maxdgram=57344 net.inet.udp.maxdgram: 9216 -> 57344 atombsd# sysctl net.inet.udp.recvspace=65536 net.inet.udp.recvspace: 42080 -> 65536 atombsd# sysctl net.local.stream.recvspace=65536 net.local.stream.recvspace: 8192 -> 65536 atombsd# sysctl net.local.stream.sendspace=65536 net.local.stream.sendspace: 8192 -> 65536 This improved the read speeds a further tiny bit, now I went from 48mb/s to 54mb/s. This is it however, I can't figure out how to increase Samba read speed any further. Any ideas? From owner-freebsd-performance@FreeBSD.ORG Sat Mar 20 01:28:05 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56EF91065673; Sat, 20 Mar 2010 01:28:05 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 5E8888FC12; Sat, 20 Mar 2010 01:28:04 +0000 (UTC) Received: by bwz8 with SMTP id 8so3538697bwz.3 for ; Fri, 19 Mar 2010 18:28:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=6r7Ruld4lQK7PdbGvYJaSwXnS0Y+83oNhcyHTfti57k=; b=YctfHydih6/5jek5ROfggr8KpAqkbnwsOJLDKJn9evraOGfuZC1uZ7OTfVHOVSZzX1 K5kWBkyCixamY9XP8Eu9zo4TRP7nU1lmlKkX9/Xyl7X0yRdrNoZaF8ux+apCwfRfNrBF ChMQzpFCxlQYAzbRg3K9jxLtKqragYzlACS3E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=jHSECJ3aaST4Vir0rdb51HRM/gAUjT7Jk6bM1VrJ9Q7OYvjOd6oIcMC+HsoPoHCeMM RjYgOWldoe2dx2byDdq6FkbnM685tJ+EuS0zwmCQR7H5vI86/8FgDP3ByRGVgpVsXatN n2v+RQiUqkCoB/johF01auN8Xy9t/GWhhB5P8= MIME-Version: 1.0 Received: by 10.204.1.136 with SMTP id 8mr1457434bkf.92.1269048482999; Fri, 19 Mar 2010 18:28:02 -0700 (PDT) In-Reply-To: References: Date: Sat, 20 Mar 2010 03:28:02 +0200 Message-ID: From: Dan Naumov To: freebsd-net@freebsd.org, freebsd-questions@freebsd.org, FreeBSD-STABLE Mailing List , freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Mailman-Approved-At: Sat, 20 Mar 2010 01:47:21 +0000 Cc: Subject: Re: Samba read speed performance tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 01:28:05 -0000 On Fri, Mar 19, 2010 at 11:14 PM, Dan Naumov wrote: > On a FreeBSD 8.0-RELEASE/amd64 system with a Supermicro X7SPA-H board > using an Intel gigabit nic with the em driver, running on top of a ZFS > mirror, I was seeing a strange issue. Local reads and writes to the > pool easily saturate the disks with roughly 75mb/s throughput, which > is roughly the best these drives can do. However, working with Samba, > writes to a share could easily pull off 75mb/s and saturate the disks, > but reads off a share were resulting in rather pathetic 18mb/s > throughput. > > I found a threadon the FreeBSD forums > (http://forums.freebsd.org/showthread.php?t=9187) and followed the > suggested advice. I rebuilt Samba with AIO support, kldloaded the aio > module and made the following changes to my smb.conf > > From: > socket options=TCP_NODELAY > > To: > socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY > min receivefile size=16384 > use sendfile=true > aio read size = 16384 > aio write size = 16384 > aio write behind = true > dns proxy = no[/CODE] > > This showed a very welcome improvement in read speed, I went from > 18mb/s to 48mb/s. The write speed remained unchanged and was still > saturating the disks. Now I tried the suggested sysctl tunables: > > atombsd# sysctl net.inet.tcp.delayed_ack=0 > net.inet.tcp.delayed_ack: 1 -> 0 > > atombsd# sysctl net.inet.tcp.path_mtu_discovery=0 > net.inet.tcp.path_mtu_discovery: 1 -> 0 > > atombsd# sysctl net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.recvbuf_inc: 16384 -> 524288 > > atombsd# sysctl net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvbuf_max: 262144 -> 16777216 > > atombsd# sysctl net.inet.tcp.sendbuf_inc=524288 > net.inet.tcp.sendbuf_inc: 8192 -> 524288 > > atombsd# sysctl net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.sendbuf_max: 262144 -> 16777216 > > atombsd# sysctl net.inet.tcp.sendspace=65536 > net.inet.tcp.sendspace: 32768 -> 65536 > > atombsd# sysctl net.inet.udp.maxdgram=57344 > net.inet.udp.maxdgram: 9216 -> 57344 > > atombsd# sysctl net.inet.udp.recvspace=65536 > net.inet.udp.recvspace: 42080 -> 65536 > > atombsd# sysctl net.local.stream.recvspace=65536 > net.local.stream.recvspace: 8192 -> 65536 > > atombsd# sysctl net.local.stream.sendspace=65536 > net.local.stream.sendspace: 8192 -> 65536 > > This improved the read speeds a further tiny bit, now I went from > 48mb/s to 54mb/s. This is it however, I can't figure out how to > increase Samba read speed any further. Any ideas? Oh my god... Why did noone tell me how much of an enormous performance boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is. My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie. they are now nearly 25% faster than reading off an individual disk) and reads off a Samba share skyrocketed from 50mb/s to 90mb/s. By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and on any amd64 systems with less than 4GB of avaiable memory. My system is amd64 with 4gb ram, but integrated video eats some of that, so the autotuning disabled the prefetch. I had read up on it and a fair amount of people seemed to have performance issues caused by having prefetch enabled and get better results with it turned off, in my case however, it seems that enabling it gave a really solid boost to performance. - Sincerely Dan Naumov From owner-freebsd-performance@FreeBSD.ORG Sat Mar 20 02:06:54 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4B25106564A; Sat, 20 Mar 2010 02:06:54 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost0.waddell.com (mailhost0.waddell.com [12.154.38.61]) by mx1.freebsd.org (Postfix) with ESMTP id 8442D8FC0C; Sat, 20 Mar 2010 02:06:54 +0000 (UTC) Received: from emlpfilt2.waddell.com (emlpfilt2.waddell.com [10.1.10.30]) by mailhost0.waddell.com (Postfix) with ESMTP id CBEF226362D; Fri, 19 Mar 2010 20:47:27 -0500 (CDT) Received: from emlpfilt2.waddell.com (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 7D21D2F8002; Fri, 19 Mar 2010 20:49:35 -0500 (CDT) Received: from advpexv0.wradvisors.com (advpexb1.wradvisors.com [192.168.203.21]) by emlpfilt2.waddell.com (Postfix) with ESMTP id 74DF42F8001; Fri, 19 Mar 2010 20:49:35 -0500 (CDT) Received: from ADVPHTCAS0.wradvisors.com ([192.168.203.228]) by advpexv0.wradvisors.com with Microsoft SMTPSVC(6.0.3790.3959); Fri, 19 Mar 2010 20:49:35 -0500 Received: from WADPMBXV0.waddell.com ([169.254.1.205]) by ADVPHTCAS0.wradvisors.com ([192.168.203.228]) with mapi; Fri, 19 Mar 2010 20:49:34 -0500 From: Gary Gatten To: "'dan.naumov@gmail.com'" , "'freebsd-net@freebsd.org'" , "'freebsd-questions@freebsd.org'" , "'freebsd-stable@freebsd.org'" , "'freebsd-performance@freebsd.org'" Date: Fri, 19 Mar 2010 20:49:34 -0500 Thread-Topic: Samba read speed performance tuning Thread-Index: AcrHzOlSisOlux8WQhGh78s4+fcm6AAAq4gk Message-ID: <30209_1269049775_4BA429AF_30209_895_1_D9B37353831173459FDAA836D3B434994A334315@WADPMBXV0.waddell.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginalArrivalTime: 20 Mar 2010 01:49:35.0203 (UTC) FILETIME=[97C6DB30:01CAC7CF] X-Mailman-Approved-At: Sat, 20 Mar 2010 02:56:14 +0000 Cc: Subject: Re: Samba read speed performance tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 02:06:55 -0000 SXQgTUFZIG1ha2UgYSBiaWcgZGlmZiwgYnV0IG1ha2Ugc3VyZSBkdXJpbmcgeW91ciB0ZXN0cyB5 b3UgdXNlIHVuaXF1ZSBmaWxlcyBvciBmbHVzaCB0aGUgY2FjaGUgb3IgeW91J2xsIG1lIHRlc3Rp bmcgY2FjaGUgc3BlZWQgYW5kIG5vdCBkaXNrIHNwZWVkLg0KDQotLS0tLSBPcmlnaW5hbCBNZXNz YWdlIC0tLS0tDQpGcm9tOiBvd25lci1mcmVlYnNkLXF1ZXN0aW9uc0BmcmVlYnNkLm9yZyA8b3du ZXItZnJlZWJzZC1xdWVzdGlvbnNAZnJlZWJzZC5vcmc+DQpUbzogZnJlZWJzZC1uZXRAZnJlZWJz ZC5vcmcgPGZyZWVic2QtbmV0QGZyZWVic2Qub3JnPjsgZnJlZWJzZC1xdWVzdGlvbnNAZnJlZWJz ZC5vcmcgPGZyZWVic2QtcXVlc3Rpb25zQGZyZWVic2Qub3JnPjsgRnJlZUJTRC1TVEFCTEUgTWFp bGluZyBMaXN0IDxmcmVlYnNkLXN0YWJsZUBmcmVlYnNkLm9yZz47IGZyZWVic2QtcGVyZm9ybWFu Y2VAZnJlZWJzZC5vcmcgPGZyZWVic2QtcGVyZm9ybWFuY2VAZnJlZWJzZC5vcmc+DQpTZW50OiBG cmkgTWFyIDE5IDIwOjI4OjAyIDIwMTAKU3ViamVjdDogUmU6IFNhbWJhIHJlYWQgc3BlZWQgcGVy Zm9ybWFuY2UgdHVuaW5nDQoNCk9uIEZyaSwgTWFyIDE5LCAyMDEwIGF0IDExOjE0IFBNLCBEYW4g TmF1bW92IDxkYW4ubmF1bW92QGdtYWlsLmNvbT4gd3JvdGU6DQo+IE9uIGEgRnJlZUJTRCA4LjAt UkVMRUFTRS9hbWQ2NCBzeXN0ZW0gd2l0aCBhIFN1cGVybWljcm8gWDdTUEEtSCBib2FyZA0KPiB1 c2luZyBhbiBJbnRlbCBnaWdhYml0IG5pYyB3aXRoIHRoZSBlbSBkcml2ZXIsIHJ1bm5pbmcgb24g dG9wIG9mIGEgWkZTDQo+IG1pcnJvciwgSSB3YXMgc2VlaW5nIGEgc3RyYW5nZSBpc3N1ZS4gTG9j YWwgcmVhZHMgYW5kIHdyaXRlcyB0byB0aGUNCj4gcG9vbCBlYXNpbHkgc2F0dXJhdGUgdGhlIGRp c2tzIHdpdGggcm91Z2hseSA3NW1iL3MgdGhyb3VnaHB1dCwgd2hpY2gNCj4gaXMgcm91Z2hseSB0 aGUgYmVzdCB0aGVzZSBkcml2ZXMgY2FuIGRvLiBIb3dldmVyLCB3b3JraW5nIHdpdGggU2FtYmEs DQo+IHdyaXRlcyB0byBhIHNoYXJlIGNvdWxkIGVhc2lseSBwdWxsIG9mZiA3NW1iL3MgYW5kIHNh dHVyYXRlIHRoZSBkaXNrcywNCj4gYnV0IHJlYWRzIG9mZiBhIHNoYXJlIHdlcmUgcmVzdWx0aW5n IGluIHJhdGhlciBwYXRoZXRpYyAxOG1iL3MNCj4gdGhyb3VnaHB1dC4NCj4NCj4gSSBmb3VuZCBh IHRocmVhZG9uIHRoZSBGcmVlQlNEIGZvcnVtcw0KPiAoaHR0cDovL2ZvcnVtcy5mcmVlYnNkLm9y Zy9zaG93dGhyZWFkLnBocD90PTkxODcpIGFuZCBmb2xsb3dlZCB0aGUNCj4gc3VnZ2VzdGVkIGFk dmljZS4gSSByZWJ1aWx0IFNhbWJhIHdpdGggQUlPIHN1cHBvcnQsIGtsZGxvYWRlZCB0aGUgYWlv DQo+IG1vZHVsZSBhbmQgbWFkZSB0aGUgZm9sbG93aW5nIGNoYW5nZXMgdG8gbXkgc21iLmNvbmYN Cj4NCj4gRnJvbToNCj4gc29ja2V0IG9wdGlvbnM9VENQX05PREVMQVkNCj4NCj4gVG86DQo+IHNv Y2tldCBvcHRpb25zPVNPX1JDVkJVRj0xMzEwNzIgU09fU05EQlVGPTEzMTA3MiBUQ1BfTk9ERUxB WQ0KPiBtaW4gcmVjZWl2ZWZpbGUgc2l6ZT0xNjM4NA0KPiB1c2Ugc2VuZGZpbGU9dHJ1ZQ0KPiBh aW8gcmVhZCBzaXplID0gMTYzODQNCj4gYWlvIHdyaXRlIHNpemUgPSAxNjM4NA0KPiBhaW8gd3Jp dGUgYmVoaW5kID0gdHJ1ZQ0KPiBkbnMgcHJveHkgPSBub1svQ09ERV0NCj4NCj4gVGhpcyBzaG93 ZWQgYSB2ZXJ5IHdlbGNvbWUgaW1wcm92ZW1lbnQgaW4gcmVhZCBzcGVlZCwgSSB3ZW50IGZyb20N Cj4gMThtYi9zIHRvIDQ4bWIvcy4gVGhlIHdyaXRlIHNwZWVkIHJlbWFpbmVkIHVuY2hhbmdlZCBh bmQgd2FzIHN0aWxsDQo+IHNhdHVyYXRpbmcgdGhlIGRpc2tzLiBOb3cgSSB0cmllZCB0aGUgc3Vn Z2VzdGVkIHN5c2N0bCB0dW5hYmxlczoNCj4NCj4gYXRvbWJzZCMgc3lzY3RsIG5ldC5pbmV0LnRj cC5kZWxheWVkX2Fjaz0wDQo+IG5ldC5pbmV0LnRjcC5kZWxheWVkX2FjazogMSAtPiAwDQo+DQo+ IGF0b21ic2QjIHN5c2N0bCBuZXQuaW5ldC50Y3AucGF0aF9tdHVfZGlzY292ZXJ5PTANCj4gbmV0 LmluZXQudGNwLnBhdGhfbXR1X2Rpc2NvdmVyeTogMSAtPiAwDQo+DQo+IGF0b21ic2QjIHN5c2N0 bCBuZXQuaW5ldC50Y3AucmVjdmJ1Zl9pbmM9NTI0Mjg4DQo+IG5ldC5pbmV0LnRjcC5yZWN2YnVm X2luYzogMTYzODQgLT4gNTI0Mjg4DQo+DQo+IGF0b21ic2QjIHN5c2N0bCBuZXQuaW5ldC50Y3Au cmVjdmJ1Zl9tYXg9MTY3NzcyMTYNCj4gbmV0LmluZXQudGNwLnJlY3ZidWZfbWF4OiAyNjIxNDQg LT4gMTY3NzcyMTYNCj4NCj4gYXRvbWJzZCMgc3lzY3RsIG5ldC5pbmV0LnRjcC5zZW5kYnVmX2lu Yz01MjQyODgNCj4gbmV0LmluZXQudGNwLnNlbmRidWZfaW5jOiA4MTkyIC0+IDUyNDI4OA0KPg0K PiBhdG9tYnNkIyBzeXNjdGwgbmV0LmluZXQudGNwLnNlbmRidWZfbWF4PTE2Nzc3MjE2DQo+IG5l dC5pbmV0LnRjcC5zZW5kYnVmX21heDogMjYyMTQ0IC0+IDE2Nzc3MjE2DQo+DQo+IGF0b21ic2Qj IHN5c2N0bCBuZXQuaW5ldC50Y3Auc2VuZHNwYWNlPTY1NTM2DQo+IG5ldC5pbmV0LnRjcC5zZW5k c3BhY2U6IDMyNzY4IC0+IDY1NTM2DQo+DQo+IGF0b21ic2QjIHN5c2N0bCBuZXQuaW5ldC51ZHAu bWF4ZGdyYW09NTczNDQNCj4gbmV0LmluZXQudWRwLm1heGRncmFtOiA5MjE2IC0+IDU3MzQ0DQo+ DQo+IGF0b21ic2QjIHN5c2N0bCBuZXQuaW5ldC51ZHAucmVjdnNwYWNlPTY1NTM2DQo+IG5ldC5p bmV0LnVkcC5yZWN2c3BhY2U6IDQyMDgwIC0+IDY1NTM2DQo+DQo+IGF0b21ic2QjIHN5c2N0bCBu ZXQubG9jYWwuc3RyZWFtLnJlY3ZzcGFjZT02NTUzNg0KPiBuZXQubG9jYWwuc3RyZWFtLnJlY3Zz cGFjZTogODE5MiAtPiA2NTUzNg0KPg0KPiBhdG9tYnNkIyBzeXNjdGwgbmV0LmxvY2FsLnN0cmVh bS5zZW5kc3BhY2U9NjU1MzYNCj4gbmV0LmxvY2FsLnN0cmVhbS5zZW5kc3BhY2U6IDgxOTIgLT4g NjU1MzYNCj4NCj4gVGhpcyBpbXByb3ZlZCB0aGUgcmVhZCBzcGVlZHMgYSBmdXJ0aGVyIHRpbnkg Yml0LCBub3cgSSB3ZW50IGZyb20NCj4gNDhtYi9zIHRvIDU0bWIvcy4gVGhpcyBpcyBpdCBob3dl dmVyLCBJIGNhbid0IGZpZ3VyZSBvdXQgaG93IHRvDQo+IGluY3JlYXNlIFNhbWJhIHJlYWQgc3Bl ZWQgYW55IGZ1cnRoZXIuIEFueSBpZGVhcz8NCg0KDQpPaCBteSBnb2QuLi4gV2h5IGRpZCBub29u ZSB0ZWxsIG1lIGhvdyBtdWNoIG9mIGFuIGVub3Jtb3VzIHBlcmZvcm1hbmNlDQpib29zdCB2ZnMu emZzLnByZWZldGNoX2Rpc2FibGU9MCAoYWthIGFjdHVhbGx5IGVuYWJsaW5nIHByZWZldGNoKSBp cy4NCk15IGxvY2FsIHJlYWRzIG9mZiB0aGUgbWlycm9yIHBvb2wganVtcGVkIGZyb20gNzVtYi9z IHRvIDk2bWIvcyAoaWUuDQp0aGV5IGFyZSBub3cgbmVhcmx5IDI1JSBmYXN0ZXIgdGhhbiByZWFk aW5nIG9mZiBhbiBpbmRpdmlkdWFsIGRpc2spDQphbmQgcmVhZHMgb2ZmIGEgU2FtYmEgc2hhcmUg c2t5cm9ja2V0ZWQgZnJvbSA1MG1iL3MgdG8gOTBtYi9zLg0KDQpCeSBkZWZhdWx0LCBGcmVlQlNE IHNldHMgdmZzLnpmcy5wcmVmZXRjaF9kaXNhYmxlIHRvIDEgb24gYW55IGkzODYNCnN5c3RlbXMg YW5kIG9uIGFueSBhbWQ2NCBzeXN0ZW1zIHdpdGggbGVzcyB0aGFuIDRHQiBvZiBhdmFpYWJsZQ0K bWVtb3J5LiBNeSBzeXN0ZW0gaXMgYW1kNjQgd2l0aCA0Z2IgcmFtLCBidXQgaW50ZWdyYXRlZCB2 aWRlbyBlYXRzDQpzb21lIG9mIHRoYXQsIHNvIHRoZSBhdXRvdHVuaW5nIGRpc2FibGVkIHRoZSBw cmVmZXRjaC4gSSBoYWQgcmVhZCB1cA0Kb24gaXQgYW5kIGEgZmFpciBhbW91bnQgb2YgcGVvcGxl IHNlZW1lZCB0byBoYXZlIHBlcmZvcm1hbmNlIGlzc3Vlcw0KY2F1c2VkIGJ5IGhhdmluZyBwcmVm ZXRjaCBlbmFibGVkIGFuZCBnZXQgYmV0dGVyIHJlc3VsdHMgd2l0aCBpdA0KdHVybmVkIG9mZiwg aW4gbXkgY2FzZSBob3dldmVyLCBpdCBzZWVtcyB0aGF0IGVuYWJsaW5nIGl0IGdhdmUgYQ0KcmVh bGx5IHNvbGlkIGJvb3N0IHRvIHBlcmZvcm1hbmNlLg0KDQoNCi0gU2luY2VyZWx5DQpEYW4gTmF1 bW92DQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KZnJl ZWJzZC1xdWVzdGlvbnNAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQpodHRwOi8vbGlzdHMuZnJl ZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLXF1ZXN0aW9ucw0KVG8gdW5zdWJzY3Jp YmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtcXVlc3Rpb25zLXVuc3Vic2NyaWJlQGZyZWVi c2Qub3JnIg0KCgoKCgo8Zm9udCBzaXplPSIxIj4KPGRpdiBzdHlsZT0nYm9yZGVyOm5vbmU7Ym9y ZGVyLWJvdHRvbTpkb3VibGUgd2luZG93dGV4dCAyLjI1cHQ7cGFkZGluZzowaW4gMGluIDEuMHB0 IDBpbic+CjwvZGl2PgoiVGhpcyBlbWFpbCBpcyBpbnRlbmRlZCB0byBiZSByZXZpZXdlZCBieSBv bmx5IHRoZSBpbnRlbmRlZCByZWNpcGllbnQKIGFuZCBtYXkgY29udGFpbiBpbmZvcm1hdGlvbiB0 aGF0IGlzIHByaXZpbGVnZWQgYW5kL29yIGNvbmZpZGVudGlhbC4KIElmIHlvdSBhcmUgbm90IHRo ZSBpbnRlbmRlZCByZWNpcGllbnQsIHlvdSBhcmUgaGVyZWJ5IG5vdGlmaWVkIHRoYXQKIGFueSBy ZXZpZXcsIHVzZSwgZGlzc2VtaW5hdGlvbiwgZGlzY2xvc3VyZSBvciBjb3B5aW5nIG9mIHRoaXMg ZW1haWwKIGFuZCBpdHMgYXR0YWNobWVudHMsIGlmIGFueSwgaXMgc3RyaWN0bHkgcHJvaGliaXRl ZC4gIElmIHlvdSBoYXZlCiByZWNlaXZlZCB0aGlzIGVtYWlsIGluIGVycm9yLCBwbGVhc2UgaW1t ZWRpYXRlbHkgbm90aWZ5IHRoZSBzZW5kZXIgYnkKIHJldHVybiBlbWFpbCBhbmQgZGVsZXRlIHRo aXMgZW1haWwgZnJvbSB5b3VyIHN5c3RlbS4iCjwvZm9udD4KCg== From owner-freebsd-performance@FreeBSD.ORG Sat Mar 20 02:57:47 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 732A9106566C for ; Sat, 20 Mar 2010 02:57:47 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 43E378FC14 for ; Sat, 20 Mar 2010 02:57:47 +0000 (UTC) Received: by pvc7 with SMTP id 7so654259pvc.13 for ; Fri, 19 Mar 2010 19:57:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=mPNlbUL+WVv1mC9T/ih8I16WRWUWrHlDRJWYTNG8TgQ=; b=Nb8gj7NroCQVlnt0obX5bTbXHiO5a5f/fDovdjyj5PTLdgfTlcB5ecHEfHGPY2P+dt qh4edUhg9NFfFXv2ze9+HQm5eOPLdmQSxLLfhDn/G9/wPDP+VZyUOXiK1hiyPDOv67k9 GLKtrD/9OD4WGPwye5mhJ+s70g52jaRjKsotM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=IID94jeo8fUzY/Vunz/G8Ypfh25dRkn+q5LhrzZLW/1v6jyzQdXXC0UhoISLVU65YO waR7opeOmmBZiTGLo3oTVXlyX1IMLJAwfm/F5CcgBnjeDs7yzvvA7zvRQ/LwcgcIppW0 4DE8Jgac68MfT4nsT6Xh653YEzNLQgjRmiSC4= MIME-Version: 1.0 Received: by 10.142.8.22 with SMTP id 22mr1435039wfh.194.1269052370909; Fri, 19 Mar 2010 19:32:50 -0700 (PDT) In-Reply-To: References: Date: Fri, 19 Mar 2010 21:32:50 -0500 Message-ID: <6201873e1003191932g447cb261rbc928324408b959b@mail.gmail.com> From: Adam Vande More To: Dan Naumov X-Mailman-Approved-At: Sat, 20 Mar 2010 04:05:45 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org, FreeBSD-STABLE Mailing List , freebsd-questions@freebsd.org Subject: Re: Samba read speed performance tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 02:57:47 -0000 On Fri, Mar 19, 2010 at 8:28 PM, Dan Naumov wrote: > Oh my god... Why did noone tell me how much of an enormous performance > boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is. > My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie. > they are now nearly 25% faster than reading off an individual disk) > and reads off a Samba share skyrocketed from 50mb/s to 90mb/s. > > By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 > systems and on any amd64 systems with less than 4GB of avaiable > memory. My system is amd64 with 4gb ram, but integrated video eats > some of that, so the autotuning disabled the prefetch. I had read up > on it and a fair amount of people seemed to have performance issues > caused by having prefetch enabled and get better results with it > turned off, in my case however, it seems that enabling it gave a > really solid boost to performance. > My home VBox server is similar specs and I enabled the prefetch from the start. A few days ago, I added an intel SSD as the zpool cache device and the read speed is mind blowing now. This is from inside a VM frunning on it meaning ad0 is really a vdi. Once the cache is populated, HD latency is mostly a thing of the past. # diskinfo -tv /dev/ad0 /dev/ad0 512 # sectorsize 12884901888 # mediasize in bytes (12G) 25165824 # mediasize in sectors 24966 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. VBf9752473-05343e4e # Disk ident. Seek times: Full stroke: 250 iter in 0.082321 sec = 0.329 msec Half stroke: 250 iter in 0.078944 sec = 0.316 msec Quarter stroke: 500 iter in 0.161266 sec = 0.323 msec Short forward: 400 iter in 0.128624 sec = 0.322 msec Short backward: 400 iter in 0.131770 sec = 0.329 msec Seq outer: 2048 iter in 0.667510 sec = 0.326 msec Seq inner: 2048 iter in 0.691691 sec = 0.338 msec Transfer rates: outside: 102400 kbytes in 0.722864 sec = 141659 kbytes/sec middle: 102400 kbytes in 0.813619 sec = 125857 kbytes/sec inside: 102400 kbytes in 0.838129 sec = 122177 kbytes/sec -- Adam Vande More From owner-freebsd-performance@FreeBSD.ORG Sat Mar 20 08:16:51 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E598A106566C; Sat, 20 Mar 2010 08:16:51 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 1E6DA8FC1C; Sat, 20 Mar 2010 08:16:50 +0000 (UTC) Received: by bwz8 with SMTP id 8so3628779bwz.3 for ; Sat, 20 Mar 2010 01:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=ZGEJNCP9LNysm18UdMBs2XqHr5ZsHJieCtGNrH0LcoY=; b=d6yZFF4iuAJrnYCvHnrgLreFc/mmy14qY9y3zEPYujFdGkTeNCIIBpDNJtsR8hmFP9 Xtkx1lgbrZUGsOFaZNIzH5Av1SOWkf9onHWlqv7xxdPiEFwWuhlESgybcVIyGPzml5gF 2b2ew853pnu2CcHTY6UBxuAxSl4ObW3XyP4iw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=lUHNQx5lYzNnKo4BVIlb7GhynV8CJu4EFvfZzY5LvNUOhrLy2iIeL3pSrSU26CkdeE Akks9LpBp5fSKAV4cObpFUubkxJeVZb9YP/fyPXXnuKLlgUBDcMu8azQxyYQdIJ44HuW r4NoxWVSSIfrMirAyh8vryhAgoWP5xM8uFUKM= MIME-Version: 1.0 Received: by 10.204.138.79 with SMTP id z15mr4003924bkt.167.1269073009288; Sat, 20 Mar 2010 01:16:49 -0700 (PDT) In-Reply-To: <30209_1269049775_4BA429AF_30209_895_1_D9B37353831173459FDAA836D3B434994A334315@WADPMBXV0.waddell.com> References: <30209_1269049775_4BA429AF_30209_895_1_D9B37353831173459FDAA836D3B434994A334315@WADPMBXV0.waddell.com> Date: Sat, 20 Mar 2010 10:16:49 +0200 Message-ID: From: Dan Naumov To: Gary Gatten Content-Type: text/plain; charset=ISO-8859-1 X-Mailman-Approved-At: Sat, 20 Mar 2010 11:29:31 +0000 Cc: "freebsd-performance@freebsd.org" , "freebsd-stable@freebsd.org" , "freebsd-questions@freebsd.org" Subject: Re: Samba read speed performance tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 08:16:52 -0000 On Sat, Mar 20, 2010 at 3:49 AM, Gary Gatten wrote: > It MAY make a big diff, but make sure during your tests you use unique files or flush the cache or you'll me testing cache speed and not disk speed. Yeah I did make sure to use unique files for testing the effects of prefetch. This is Atom D510 / Supermicro X75SPA-H / 4Gb Ram with 2 x slow 2tb WD Green (WD20EADS) disks with 32mb cache in a ZFS mirror after enabling prefetch.: Code: bonnie -s 8192 -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 8192 29065 68.9 52027 39.8 39636 33.3 54057 95.4 105335 34.6 174.1 7.9 DD read: dd if=/dev/urandom of=test2 bs=1M count=8192 dd if=test2 of=/dev/zero bs=1M 8589934592 bytes transferred in 76.031399 secs (112978779 bytes/sec) (107,74mb/s) Individual disks read capability: 75mb/s Reading off a mirror of 2 disks with prefetch disabled: 60mb/s Reading off a mirror of 2 disks with prefetch enabled: 107mb/s - Sincerely, Dan Naumov