From owner-freebsd-fs@freebsd.org Sun Jan 3 16:13:15 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A26DA607B3 for ; Sun, 3 Jan 2016 16:13:15 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F4C01299 for ; Sun, 3 Jan 2016 16:13:14 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id u03G9CO4016169; Sun, 3 Jan 2016 10:09:12 -0600 (CST) Date: Sun, 3 Jan 2016 10:09:12 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: "Mikhail T." cc: freebsd-fs@freebsd.org Subject: Re: NFS reads vs. writes In-Reply-To: <568880D3.3010402@aldan.algebra.com> Message-ID: References: <568880D3.3010402@aldan.algebra.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sun, 03 Jan 2016 10:09:12 -0600 (CST) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jan 2016 16:13:15 -0000 On Sat, 2 Jan 2016, Mikhail T. wrote: > I have two similar 10.2-STABLE machines mounting each other's > filesystems via NFS: a:/a and b:/b > > When moving some large files around today, I found a huge discrepancy > between writing and reading over NFS. That is, machine a copying files > from its own /a to NFS-mounted b:/b was pushing measly 2-3Mb/s over the > Ethernet. When I cancelled that, logged-in to machine b and proceed to > copy from a:/a to the now-local /b, I got 56Mb/s. I tried changing the > wsize and rsize options, but it did not seem to make a difference... > > Any ideas, what is happening here? Why are NFS-writes some 25 times > slower here, than reads? Both filesystems are zfs-backed (though with > different options), both systems are plugged into the same switch, both > use mtu of 9000. The most likely issue is a latency problem with synchronous writes on 'b'. The main pool disks seem to be working ok. Make sure that the SSD you are using for slog is working fine. Maybe it is abnormally slow. Check I/O latencies for your disks. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/