From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 20 05:27:08 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74575106566C for ; Sun, 20 Dec 2009 05:27:08 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email1.allantgroup.com (email1.emsphone.com [199.67.51.115]) by mx1.freebsd.org (Postfix) with ESMTP id 3A80A8FC5A for ; Sun, 20 Dec 2009 05:27:07 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email1.allantgroup.com (8.14.0/8.14.0) with ESMTP id nBK5R50l015811 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 19 Dec 2009 23:27:05 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id nBK5R5BB071720 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 19 Dec 2009 23:27:05 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.3/8.14.3/Submit) id nBK5R4rs071716; Sat, 19 Dec 2009 23:27:04 -0600 (CST) (envelope-from dan) Date: Sat, 19 Dec 2009 23:27:04 -0600 From: Dan Nelson To: Zaphod Beeblebrox Message-ID: <20091220052703.GA98917@dan.emsphone.com> References: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> X-OS: FreeBSD 7.2-STABLE User-Agent: Mutt/1.5.20 (2009-06-14) X-Virus-Scanned: ClamAV version 0.94.1, clamav-milter version 0.94.1 on email1.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (email1.allantgroup.com [199.67.51.78]); Sat, 19 Dec 2009 23:27:05 -0600 (CST) X-Scanned-By: MIMEDefang 2.45 Cc: freebsd-hackers@freebsd.org Subject: Re: scp more perfectly fills the pipe than NFS/TCP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Dec 2009 05:27:08 -0000 In the last episode (Dec 19), Zaphod Beeblebrox said: > Here's an interesting conundrum. I don't know what's different between > the TCP that scp uses from the TCP that NFS uses, but given the same two > FreeBSD machines, SCP fills the pipe with packets better. > > Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrtg.png > > The system doing the scp and the NFS server is FreeBSD-7.2-p1. The system > receiving the scp and the NFS client is FreeBSD-8.0-p1 > > The scp transfer is the left hand side of the graph and the NFS transfer > is on the right. > > The NFS is mounted with "-3 -T -b -l -i" and no other options. Files are > being moved over NFS with the system "mv" command. The files in each case > are large (50 to 500 meg files). If you increase the NFS blocksize (-r 32768 for example) you will get slightly better performance, but you will likely never match the scp results. They're doing two different things under the hood: scp is streaming the entire file in one operation, while NFS is performing many "read 8k at offset 0", "read 8k at offset 8k", etc requests one after another, so a high-latency connection will take a performance hit due to the latency in issuing each command. According to the mount_nfs manpage, it looks like there is some prefetching that can be enabled with the "-a ##" option. It doesn't say what the default is, though. -- Dan Nelson dnelson@allantgroup.com