From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 21 05:47:57 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA8FA1065672 for ; Mon, 21 Dec 2009 05:47:57 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.24]) by mx1.freebsd.org (Postfix) with ESMTP id 4F23B8FC18 for ; Mon, 21 Dec 2009 05:47:57 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 4so1249766eyf.9 for ; Sun, 20 Dec 2009 21:47:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=kHLVqXGa5csLmaySkdV49t1U25XkTZf1EHnY7Bj9s1c=; b=qHyUTr3N/dXGDJsLm7nfO5wBiCEsU9X9JQpbKEKzCVi8KvXNVcgF9nhKk4WYIJlfOE Oi82hHHsn8ifQ2DKYE9SyqJ5bSE1SSPFWDeLKKA4plr1AuEf38nkmGmzH08uRZ7Awvfi fUaITKza1O68Io3kGqF3aNeqtyQg1g4go62o8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=XpwRtCUhmC+pXyPDslJWQ7H0VLLRACJOVLD0X2/uZzp8vE7z9znooCNgcYrkTXfEC9 ONGURmfAQMZwQbkJy27lVVVzK8ReSgK7FaqEejiWlFBDiBg2FwdobkaYDBOhDz75IjIK ymJeSwh4evxZXYlKXKRrX/QstgR3F8685tYmU= MIME-Version: 1.0 Received: by 10.216.163.81 with SMTP id z59mr2620262wek.95.1261374475966; Sun, 20 Dec 2009 21:47:55 -0800 (PST) In-Reply-To: <20091220052703.GA98917@dan.emsphone.com> References: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> <20091220052703.GA98917@dan.emsphone.com> Date: Mon, 21 Dec 2009 00:47:55 -0500 Message-ID: <5f67a8c40912202147t9d9b64al88060bd8a73c28b0@mail.gmail.com> From: Zaphod Beeblebrox To: Dan Nelson Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: scp more perfectly fills the pipe than NFS/TCP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Dec 2009 05:47:57 -0000 On Sun, Dec 20, 2009 at 12:27 AM, Dan Nelson wrot= e: > In the last episode (Dec 19), Zaphod Beeblebrox said: >> Here's an interesting conundrum. =A0I don't know what's different betwee= n >> the TCP that scp uses from the TCP that NFS uses, but given the same two >> FreeBSD machines, SCP fills the pipe with packets better. >> >> Examine the following graphic: http://www.eicat.ca/~dgilbert/example-mrt= g.png >> >> The system doing the scp and the NFS server is FreeBSD-7.2-p1. =A0The sy= stem >> receiving the scp and the NFS client is FreeBSD-8.0-p1 >> >> The scp transfer is the left hand side of the graph and the NFS transfer >> is on the right. >> >> The NFS is mounted with "-3 -T -b -l -i" and no other options. =A0Files = are >> being moved over NFS with the system "mv" command. =A0The files in each = case >> are large (50 to 500 meg files). > > If you increase the NFS blocksize (-r 32768 for example) you will get > slightly better performance, but you will likely never match the scp > results. =A0They're doing two different things under the hood: scp is > streaming the entire file in one operation, while NFS is performing many > "read 8k at offset 0", "read 8k at offset 8k", etc requests one after > another, so a high-latency connection will take a performance hit due to = the > latency in issuing each command. =A0According to the mount_nfs manpage, i= t > looks like there is some prefetching that can be enabled with the "-a ##" > option. =A0It doesn't say what the default is, though. While the link is slow, it is really directly connected with a latency of 10ms or so. Isn't mv mmap()'ing large enough regions to cause there to be a reasonable queue to transfer?