Date: Mon, 4 Jan 2016 18:58:16 -0500 From: Tom Curry <thomasrcurry@gmail.com> To: "Mikhail T." <mi+thun@aldan.algebra.com> Cc: Rick Macklem <rmacklem@uoguelph.ca>, freebsd-fs@freebsd.org Subject: Re: NFS reads vs. writes Message-ID: <CAGtEZUD28UZDYyHtHtzXgys%2Brpv_37u4fotwR%2BqZLc1%2BtK0dwA@mail.gmail.com> In-Reply-To: <568A047B.1010000@aldan.algebra.com> References: <8291bb85-bd01-4c8c-80f7-2adcf9947366@email.android.com> <5688D3C1.90301@aldan.algebra.com> <495055121.147587416.1451871433217.JavaMail.zimbra@uoguelph.ca> <568A047B.1010000@aldan.algebra.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jan 4, 2016 at 12:34 AM, Mikhail T. <mi+thun@aldan.algebra.com> wrote: > On 03.01.2016 20:37, Rick Macklem wrote: > > This issue isn't new. It showed up when Sun introduced NFS in 1985. > > NFSv3 did change things a little, by allowing UNSTABLE writes. > Thank you very much, Rick, for the detailed explanation. > > If you use "sync=disabled" > > (I'm not a ZFS guy, but I think that is what the ZFS option looks > likes) you > > *break* the NFS protocol (ie. violate the RFC) and put your data > at some risk, > > but you will typically get better (often much better) write > performance. > Yes, indeed. Disabling sync got the writing throughput all the way up to > about 86Mb/s... I still don't fully understand, why local writes are > able to achieve this speed without async and without being considered > dangerous. > > Also, the NFS server was recently tweaked so that it could handle 128K > rsize/wsize, > > but the FreeBSD client is limited to MAXBSIZE and this has not been > increased > > beyond 64K. > I just tried lowering ZFS' recordsize to 64k to match MAXBSIZE, but that > didn't help NFS-writing (unless sync is disabled, that is). > > If this SSD is dedicated to the ZIL and is one known to have good write > performance, > > it should help, but in your case the SSD seems to be the bottleneck. > It is a chunk of an older SSD, that also houses the OS. But it is > usually idle, because executables and libraries are cached in the > abundant RAM. I've seen it do 90+Mb/s (sequential)... > > I just tried removing ZIL from the receiving pool -- to force direct > writes -- but it didn't help the case, where the writes go over NFS. > I assume you mean you removed the SLOG from the pool, in which case you most definitely still have a ZIL, its now located on the pool itself. Assuming you still have sync=standard I would venture a guess that writes over NFS would now be measured in KB/s. > However, the local writes -- with reads from NFS -- went from the 56Mb/s > I was seeing earlier to 90Mb/s!.. > > There is got to be a better way to do this -- preferably, some > self-tuning smarts... Thanks again. Yours, > > There is no getting around the performance impact of a synchronous operation, whether its NFS or a database log. If you don't believe me hop on your favorite Windows box, bring up the device manager and disable the write cache on its drive then run some benchmark supporting sync writes. One way to lessen the performance impact is to decrease the latency of writes, which is why SSD SLOGs help so much. Which brings me to my next point.. SSDs are so fast for three main reasons: low latency, large dram buffers, and parallel workloads. Only one of these is of any benefit (latency) as a SLOG. Unfortunately that particular metric is not usually advertised in consumer SSDs where the benchmarks they use to tout 90,000 random write iops consist of massively concurrent, highly compressible, short lived bursts of data. Add that drive as a SLOG and the onboard dram may as well not even exist, and queue depths count for nothing. It will be lucky to pull 2,000 IOPS. Once you start adding in ZFS features like checksums and compression, or network latency in the case of NFS that 2,000 number starts to drop even more. If you fancy a deeper understanding, this excellent article will take you as far down the rabbit hole as you wish to go. It's very informative. http://dtrace.org/blogs/brendan/2009/06/26/slog-screenshots/ > -mi > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGtEZUD28UZDYyHtHtzXgys%2Brpv_37u4fotwR%2BqZLc1%2BtK0dwA>