From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 21 20:53:47 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 624E71065676 for ; Mon, 21 Dec 2009 20:53:47 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 07AFA8FC18 for ; Mon, 21 Dec 2009 20:53:46 +0000 (UTC) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.2/8.14.1) with ESMTP id nBLKgb6Y021551; Mon, 21 Dec 2009 12:42:37 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.14.2/8.13.4/Submit) id nBLKga0G021550; Mon, 21 Dec 2009 12:42:36 -0800 (PST) Date: Mon, 21 Dec 2009 12:42:36 -0800 (PST) From: Matthew Dillon Message-Id: <200912212042.nBLKga0G021550@apollo.backplane.com> To: Zaphod Beeblebrox References: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: scp more perfectly fills the pipe than NFS/TCP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Dec 2009 20:53:47 -0000 Play with the read-ahead mount options for NFS, but it might require more work with that kind of latency. You need to be able to have a lot of RPC's in-flight to maintain the pipeline with higher latencies. At least 16 and possibly more. It might be easier to investigate why the latency is so high and fix that first. 10ms is way too high for a LAN. I remember there was some work in the FreeBSD tree to clean up the client-side NFS rpc mechanics but if they are still threaded (kernel thread or user thread, doesn't matter) with one synchronous RPC per thread then a large amount of read-ahead will cause the requests to be issued out of order over the wire (for both TCP and UDP NFS mounts), which really messes up the server-side heuristics. Plus the client-side threads wind up competing with each other for the socket lock. So there is a limit to how large a read-ahead you can specify and still get good results. If they are using a single kernel thread for socket reading and a single kernel thread for socket writing (i.e. a 100% async RPC model, which is what DFly uses), then you can boost the read-ahead to 50+. At that point the socket buffer becomes the limiting factor in the pipeline. Make sure the NFS mount is TCP (It defaults to TCP in FreeBSD 8+). UDP mounts will not perform well with any read-ahead greater then 3 or 4 RPCs because occassional seek latencies on the server will cause random UDP RPCs to timeout and retry, which completely destroys performance. UDP mounts have no understanding of the RPC queue backlog on the server and treat each RPC independently for timeout/retry purposes. So one minor stall can blow up every single pending RPC backed up behind the one that stalled. -Matt