From owner-freebsd-fs@FreeBSD.ORG Wed Aug 28 15:11:12 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AB82EB4B for ; Wed, 28 Aug 2013 15:11:12 +0000 (UTC) (envelope-from aurfalien@gmail.com) Received: from mail-oa0-x22f.google.com (mail-oa0-x22f.google.com [IPv6:2607:f8b0:4003:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 75ADB2F9A for ; Wed, 28 Aug 2013 15:11:12 +0000 (UTC) Received: by mail-oa0-f47.google.com with SMTP id g12so7812986oah.20 for ; Wed, 28 Aug 2013 08:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=lOpwCcyuzKTTNo0umqV1XCzZik8D7kau0CN1gYQYZ18=; b=nJ45Dy6AUpxDknVbbrGLnejZGjxmIRtX36ub48Kwlt5eC2Ygrm62e3qGMq2w0UtGTs 7CAP4SdqDjfFaXGwXYrCQm9LQfuPwuny17qtqRWi7pTFknRRA4KVuAK0QkVAaBPfPWE0 58P77AfSHMhczq1r068Ni0vk2ntHPSfzB6We9+/3ew7lWJQa1QOFoO5faCP2Vse5QE78 sKAcfUxi+QZE+PMxjjMzf4S2WEVzFZP7EUpWGyQdwFCMyauqK7VInrYjqebhzhxCeRsh pHYGHZ74tX8j2S42eURXq32Z9PfA1GMRhGSKcU9C+z8EYlTmbkxAoZxJh3WzkJAOmpcU Al1A== X-Received: by 10.60.96.131 with SMTP id ds3mr6902880oeb.50.1377702671841; Wed, 28 Aug 2013 08:11:11 -0700 (PDT) Received: from [192.168.1.74] (75-63-29-182.lightspeed.irvnca.sbcglobal.net. [75.63.29.182]) by mx.google.com with ESMTPSA id ps5sm27121195oeb.8.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 28 Aug 2013 08:11:10 -0700 (PDT) Subject: Re: NFS on ZFS pure SSD pool Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: aurfalien In-Reply-To: <220720097.14453943.1377644890208.JavaMail.root@uoguelph.ca> Date: Wed, 28 Aug 2013 08:11:08 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <220720097.14453943.1377644890208.JavaMail.root@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.1085) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2013 15:11:12 -0000 On Aug 27, 2013, at 4:08 PM, Rick Macklem wrote: > I wrote: >> Outback Dingo wrote: >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> On Tue, Aug 27, 2013 at 3:29 PM, Rick Macklem < >>> rmacklem@uoguelph.ca >>>> wrote: >>>=20 >>>=20 >>>=20 >>>=20 >>> Eric Browning wrote: >>>> Hello, first time posting to this list. I have a new server that >>>> is >>>> not >>>> living up to the promise of SSD speeds and NFS is maxing out the >>>> CPU. >>>> I'm >>>> new to FreeBSD but I've been reading up on it as much as I can. I >>>> have >>>> obscured my IP addresses and hostname with x's so just ignore >>>> that. >>>> Server has about 200 users on it each draing under 50Mb/s peak >>>> sustained >>>> around 1-2Mb/s >>>>=20 >>>> I've followed some network tuning guides for our I350t4 nic and >>>> that >>>> has >>>> helped with network performance somewhat but the server is still >>>> experiencing heavy load with pegging the CPU at 1250% on average >>>> with >>>> only >>>> 50Mb/s of traffin in/out of the machine. All of the network >>>> tuning >>>> came >>>> from https://calomel.org/freebsd_network_tuning.html since it was >>>> relevant >>>> to the same nic that I have. >>>>=20 >>>> Server Specs: >>>> FreeBSD 9.1 >>>> 16 cores AMDx64 >>>> 64GB of ram >>>> ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe >>>> Intel DC S3500 for ZIL and enabling/disabling has made no >>>> difference >>>> Used a spare DC S3700 for the ZIL and that made no difference >>>> either. >>>> NFS v3 & v4 for Mac home folders whose Cache fodler is >>>> redirected. >>>>=20 >>>> I've tried: >>>> Compression on/of <-- no appreciable difference >>>> Deduplication on/off <-- no appreciable difference >>>> sync=3Ddisabled and sync=3Dstandard <-- no appreciable difference >>>> setting arc cache to 56GB and also to 32GB <-- no difference in >>>> performance >>>> in terms of kern. >>>>=20 >>>> I've tried to follow the freebsd tuning guide: >>>> https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've >>>> read >>>> everything I can find on NFS on ZFS and nothing has helped. WHere >>>> am >>>> I >>>> going wrong? >>>>=20 >>> You could try this patch: >>> http://people.freebsd.org/~rmacklem/drc4-stable9.patch >>> - After applying the patch and booting a kernel built from the >>> patched >>> sources, you need to increase the value of vfs.nfsd.tcphighwater. >>> (Try something like 5000 for it as a starting point.) >>>=20 >>>=20 >>>=20 >>>=20 >>> can we get a brief on what this is supposed to improve upon ? >>>=20 >> It was developed for and tested by wollman@ to reduce mutex lock >> contention and CPU overheads for the duplicate request cache, mainly >> for NFS over TCP. (For the CPU overheads case, it allows the cache >> to grow larger, reducing the frequency and, therefore, overhead of >> trimming out stale entries.) > Oh, and I should also mention that ivoras@ developed a similar patch = which > had better code structure than mine. I did use some of his code in the = patch > that went into head, but not as much as I would have liked, because I = wanted > to get it into head before code slush for 10.0. (I had already missed = the 9.2 > release.) I take it that this patch can be applied to 9.2 when its officially = released? You mentioned=20 - aurf=