From owner-freebsd-fs@freebsd.org Mon Jan 4 06:35:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3465FA5F12F for ; Mon, 4 Jan 2016 06:35:40 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from vms173013pub.verizon.net (vms173013pub.verizon.net [206.46.173.13]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 177C41589 for ; Mon, 4 Jan 2016 06:35:39 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from aldan.narawntapu ([100.1.236.52]) by vms173013.mailsrvcs.net (Oracle Communications Messaging Server 7.0.5.32.0 64bit (built Jul 16 2014)) with ESMTPA id <0O0E00J9PY637U40@vms173013.mailsrvcs.net> for freebsd-fs@FreeBSD.org; Sun, 03 Jan 2016 23:34:56 -0600 (CST) X-CMAE-Score: 0 X-CMAE-Analysis: v=2.1 cv=btqxfxui c=1 sm=1 tr=0 a=UorMnhrCY2jH/mPejITChw==:117 a=LaogzpLLAAAA:8 a=oR5dmqMzAAAA:8 a=7aQ_Q-yQQ-AA:10 a=r77TgQKjGQsHNAKrUKIA:9 a=x5da3wTigIZacOizYuAA:9 a=QEXdDO2ut3YA:10 a=enqpX9APRVFHdJLyDM4A:9 a=sVClx91riHHoF91-:21 a=_W_S_7VecoQA:10 Subject: Re: NFS reads vs. writes To: Rick Macklem References: <8291bb85-bd01-4c8c-80f7-2adcf9947366@email.android.com> <5688D3C1.90301@aldan.algebra.com> <495055121.147587416.1451871433217.JavaMail.zimbra@uoguelph.ca> Cc: =?UTF-8?Q?Karli_Sj=c3=b6berg?= , freebsd-fs@FreeBSD.org From: "Mikhail T." X-Enigmail-Draft-Status: N1110 Message-id: <568A047B.1010000@aldan.algebra.com> Date: Mon, 04 Jan 2016 00:34:51 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-version: 1.0 In-reply-to: <495055121.147587416.1451871433217.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jan 2016 06:35:40 -0000 On 03.01.2016 20:37, Rick Macklem wrote: > This issue isn't new. It showed up when Sun introduced NFS in 1985. > NFSv3 did change things a little, by allowing UNSTABLE writes. Thank you very much, Rick, for the detailed explanation. > If you use "sync=disabled" > (I'm not a ZFS guy, but I think that is what the ZFS option looks likes) you > *break* the NFS protocol (ie. violate the RFC) and put your data at some risk, > but you will typically get better (often much better) write performance. Yes, indeed. Disabling sync got the writing throughput all the way up to about 86Mb/s... I still don't fully understand, why local writes are able to achieve this speed without async and without being considered dangerous. > Also, the NFS server was recently tweaked so that it could handle 128K rsize/wsize, > but the FreeBSD client is limited to MAXBSIZE and this has not been increased > beyond 64K. I just tried lowering ZFS' recordsize to 64k to match MAXBSIZE, but that didn't help NFS-writing (unless sync is disabled, that is). > If this SSD is dedicated to the ZIL and is one known to have good write performance, > it should help, but in your case the SSD seems to be the bottleneck. It is a chunk of an older SSD, that also houses the OS. But it is usually idle, because executables and libraries are cached in the abundant RAM. I've seen it do 90+Mb/s (sequential)... I just tried removing ZIL from the receiving pool -- to force direct writes -- but it didn't help the case, where the writes go over NFS. However, the local writes -- with reads from NFS -- went from the 56Mb/s I was seeing earlier to 90Mb/s!.. There is got to be a better way to do this -- preferably, some self-tuning smarts... Thanks again. Yours, -mi