From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 21 07:27:45 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4BEE106568B for ; Mon, 21 Dec 2009 07:27:45 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email1.allantgroup.com (email1.emsphone.com [199.67.51.115]) by mx1.freebsd.org (Postfix) with ESMTP id 685258FC08 for ; Mon, 21 Dec 2009 07:27:45 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email1.allantgroup.com (8.14.0/8.14.0) with ESMTP id nBL7RiNc070066 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 21 Dec 2009 01:27:44 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id nBL7RivG095477 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 21 Dec 2009 01:27:44 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.3/8.14.3/Submit) id nBL7RisO095476; Mon, 21 Dec 2009 01:27:44 -0600 (CST) (envelope-from dan) Date: Mon, 21 Dec 2009 01:27:44 -0600 From: Dan Nelson To: Zaphod Beeblebrox Message-ID: <20091221072743.GD98917@dan.emsphone.com> References: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> <20091220052703.GA98917@dan.emsphone.com> <5f67a8c40912202147t9d9b64al88060bd8a73c28b0@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5f67a8c40912202147t9d9b64al88060bd8a73c28b0@mail.gmail.com> X-OS: FreeBSD 7.2-STABLE User-Agent: Mutt/1.5.20 (2009-06-14) X-Virus-Scanned: ClamAV version 0.94.1, clamav-milter version 0.94.1 on email1.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (email1.allantgroup.com [199.67.51.78]); Mon, 21 Dec 2009 01:27:44 -0600 (CST) X-Scanned-By: MIMEDefang 2.45 Cc: freebsd-hackers@freebsd.org Subject: Re: scp more perfectly fills the pipe than NFS/TCP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Dec 2009 07:27:45 -0000 In the last episode (Dec 21), Zaphod Beeblebrox said: > On Sun, Dec 20, 2009 at 12:27 AM, Dan Nelson wrote: > > In the last episode (Dec 19), Zaphod Beeblebrox said: > >> Here's an interesting conundrum.  I don't know what's different between > >> the TCP that scp uses from the TCP that NFS uses, but given the same > >> two FreeBSD machines, SCP fills the pipe with packets better. > >> > >> Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrtg.png > >> > >> The system doing the scp and the NFS server is FreeBSD-7.2-p1.  The > >> system receiving the scp and the NFS client is FreeBSD-8.0-p1 > >> > >> The scp transfer is the left hand side of the graph and the NFS > >> transfer is on the right. > >> > >> The NFS is mounted with "-3 -T -b -l -i" and no other options.  Files > >> are being moved over NFS with the system "mv" command.  The files in > >> each case are large (50 to 500 meg files). > > > > If you increase the NFS blocksize (-r 32768 for example) you will get > > slightly better performance, but you will likely never match the scp > > results.  They're doing two different things under the hood: scp is > > streaming the entire file in one operation, while NFS is performing many > > "read 8k at offset 0", "read 8k at offset 8k", etc requests one after > > another, so a high-latency connection will take a performance hit due to > > the latency in issuing each command.  According to the mount_nfs > > manpage, it looks like there is some prefetching that can be enabled > > with the "-a ##" option.  It doesn't say what the default is, though. > > While the link is slow, it is really directly connected with a latency > of 10ms or so. Isn't mv mmap()'ing large enough regions to cause > there to be a reasonable queue to transfer? I've never been impressed with FreeBSD's ability to detect sequential read patterns and prefetch blocks ahead of time, even on local ufs filesystems. -- Dan Nelson dnelson@allantgroup.com