From owner-freebsd-questions@FreeBSD.ORG Sat Mar 20 01:28:05 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56EF91065673; Sat, 20 Mar 2010 01:28:05 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 5E8888FC12; Sat, 20 Mar 2010 01:28:04 +0000 (UTC) Received: by bwz8 with SMTP id 8so3538697bwz.3 for ; Fri, 19 Mar 2010 18:28:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=6r7Ruld4lQK7PdbGvYJaSwXnS0Y+83oNhcyHTfti57k=; b=YctfHydih6/5jek5ROfggr8KpAqkbnwsOJLDKJn9evraOGfuZC1uZ7OTfVHOVSZzX1 K5kWBkyCixamY9XP8Eu9zo4TRP7nU1lmlKkX9/Xyl7X0yRdrNoZaF8ux+apCwfRfNrBF ChMQzpFCxlQYAzbRg3K9jxLtKqragYzlACS3E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=jHSECJ3aaST4Vir0rdb51HRM/gAUjT7Jk6bM1VrJ9Q7OYvjOd6oIcMC+HsoPoHCeMM RjYgOWldoe2dx2byDdq6FkbnM685tJ+EuS0zwmCQR7H5vI86/8FgDP3ByRGVgpVsXatN n2v+RQiUqkCoB/johF01auN8Xy9t/GWhhB5P8= MIME-Version: 1.0 Received: by 10.204.1.136 with SMTP id 8mr1457434bkf.92.1269048482999; Fri, 19 Mar 2010 18:28:02 -0700 (PDT) In-Reply-To: References: Date: Sat, 20 Mar 2010 03:28:02 +0200 Message-ID: From: Dan Naumov To: freebsd-net@freebsd.org, freebsd-questions@freebsd.org, FreeBSD-STABLE Mailing List , freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: Samba read speed performance tuning X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 01:28:05 -0000 On Fri, Mar 19, 2010 at 11:14 PM, Dan Naumov wrote: > On a FreeBSD 8.0-RELEASE/amd64 system with a Supermicro X7SPA-H board > using an Intel gigabit nic with the em driver, running on top of a ZFS > mirror, I was seeing a strange issue. Local reads and writes to the > pool easily saturate the disks with roughly 75mb/s throughput, which > is roughly the best these drives can do. However, working with Samba, > writes to a share could easily pull off 75mb/s and saturate the disks, > but reads off a share were resulting in rather pathetic 18mb/s > throughput. > > I found a threadon the FreeBSD forums > (http://forums.freebsd.org/showthread.php?t=9187) and followed the > suggested advice. I rebuilt Samba with AIO support, kldloaded the aio > module and made the following changes to my smb.conf > > From: > socket options=TCP_NODELAY > > To: > socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY > min receivefile size=16384 > use sendfile=true > aio read size = 16384 > aio write size = 16384 > aio write behind = true > dns proxy = no[/CODE] > > This showed a very welcome improvement in read speed, I went from > 18mb/s to 48mb/s. The write speed remained unchanged and was still > saturating the disks. Now I tried the suggested sysctl tunables: > > atombsd# sysctl net.inet.tcp.delayed_ack=0 > net.inet.tcp.delayed_ack: 1 -> 0 > > atombsd# sysctl net.inet.tcp.path_mtu_discovery=0 > net.inet.tcp.path_mtu_discovery: 1 -> 0 > > atombsd# sysctl net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.recvbuf_inc: 16384 -> 524288 > > atombsd# sysctl net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvbuf_max: 262144 -> 16777216 > > atombsd# sysctl net.inet.tcp.sendbuf_inc=524288 > net.inet.tcp.sendbuf_inc: 8192 -> 524288 > > atombsd# sysctl net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.sendbuf_max: 262144 -> 16777216 > > atombsd# sysctl net.inet.tcp.sendspace=65536 > net.inet.tcp.sendspace: 32768 -> 65536 > > atombsd# sysctl net.inet.udp.maxdgram=57344 > net.inet.udp.maxdgram: 9216 -> 57344 > > atombsd# sysctl net.inet.udp.recvspace=65536 > net.inet.udp.recvspace: 42080 -> 65536 > > atombsd# sysctl net.local.stream.recvspace=65536 > net.local.stream.recvspace: 8192 -> 65536 > > atombsd# sysctl net.local.stream.sendspace=65536 > net.local.stream.sendspace: 8192 -> 65536 > > This improved the read speeds a further tiny bit, now I went from > 48mb/s to 54mb/s. This is it however, I can't figure out how to > increase Samba read speed any further. Any ideas? Oh my god... Why did noone tell me how much of an enormous performance boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is. My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie. they are now nearly 25% faster than reading off an individual disk) and reads off a Samba share skyrocketed from 50mb/s to 90mb/s. By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and on any amd64 systems with less than 4GB of avaiable memory. My system is amd64 with 4gb ram, but integrated video eats some of that, so the autotuning disabled the prefetch. I had read up on it and a fair amount of people seemed to have performance issues caused by having prefetch enabled and get better results with it turned off, in my case however, it seems that enabling it gave a really solid boost to performance. - Sincerely Dan Naumov