Date: Wed, 28 Aug 2013 08:11:08 -0700 From: aurfalien <aurfalien@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: NFS on ZFS pure SSD pool Message-ID: <AE2DE114-5ABE-4BD0-8E08-8961BB25BAB2@gmail.com> In-Reply-To: <220720097.14453943.1377644890208.JavaMail.root@uoguelph.ca> References: <220720097.14453943.1377644890208.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 27, 2013, at 4:08 PM, Rick Macklem wrote: > I wrote: >> Outback Dingo wrote: >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> On Tue, Aug 27, 2013 at 3:29 PM, Rick Macklem < >>> rmacklem@uoguelph.ca >>>> wrote: >>>=20 >>>=20 >>>=20 >>>=20 >>> Eric Browning wrote: >>>> Hello, first time posting to this list. I have a new server that >>>> is >>>> not >>>> living up to the promise of SSD speeds and NFS is maxing out the >>>> CPU. >>>> I'm >>>> new to FreeBSD but I've been reading up on it as much as I can. I >>>> have >>>> obscured my IP addresses and hostname with x's so just ignore >>>> that. >>>> Server has about 200 users on it each draing under 50Mb/s peak >>>> sustained >>>> around 1-2Mb/s >>>>=20 >>>> I've followed some network tuning guides for our I350t4 nic and >>>> that >>>> has >>>> helped with network performance somewhat but the server is still >>>> experiencing heavy load with pegging the CPU at 1250% on average >>>> with >>>> only >>>> 50Mb/s of traffin in/out of the machine. All of the network >>>> tuning >>>> came >>>> from https://calomel.org/freebsd_network_tuning.html since it was >>>> relevant >>>> to the same nic that I have. >>>>=20 >>>> Server Specs: >>>> FreeBSD 9.1 >>>> 16 cores AMDx64 >>>> 64GB of ram >>>> ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe >>>> Intel DC S3500 for ZIL and enabling/disabling has made no >>>> difference >>>> Used a spare DC S3700 for the ZIL and that made no difference >>>> either. >>>> NFS v3 & v4 for Mac home folders whose Cache fodler is >>>> redirected. >>>>=20 >>>> I've tried: >>>> Compression on/of <-- no appreciable difference >>>> Deduplication on/off <-- no appreciable difference >>>> sync=3Ddisabled and sync=3Dstandard <-- no appreciable difference >>>> setting arc cache to 56GB and also to 32GB <-- no difference in >>>> performance >>>> in terms of kern. >>>>=20 >>>> I've tried to follow the freebsd tuning guide: >>>> https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've >>>> read >>>> everything I can find on NFS on ZFS and nothing has helped. WHere >>>> am >>>> I >>>> going wrong? >>>>=20 >>> You could try this patch: >>> http://people.freebsd.org/~rmacklem/drc4-stable9.patch >>> - After applying the patch and booting a kernel built from the >>> patched >>> sources, you need to increase the value of vfs.nfsd.tcphighwater. >>> (Try something like 5000 for it as a starting point.) >>>=20 >>>=20 >>>=20 >>>=20 >>> can we get a brief on what this is supposed to improve upon ? >>>=20 >> It was developed for and tested by wollman@ to reduce mutex lock >> contention and CPU overheads for the duplicate request cache, mainly >> for NFS over TCP. (For the CPU overheads case, it allows the cache >> to grow larger, reducing the frequency and, therefore, overhead of >> trimming out stale entries.) > Oh, and I should also mention that ivoras@ developed a similar patch = which > had better code structure than mine. I did use some of his code in the = patch > that went into head, but not as much as I would have liked, because I = wanted > to get it into head before code slush for 10.0. (I had already missed = the 9.2 > release.) I take it that this patch can be applied to 9.2 when its officially = released? You mentioned=20 - aurf=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AE2DE114-5ABE-4BD0-8E08-8961BB25BAB2>