From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 19 10:33:21 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4314616A4B3 for ; Fri, 19 Sep 2003 10:33:21 -0700 (PDT) Received: from pimout4-ext.prodigy.net (pimout4-ext.prodigy.net [207.115.63.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 218B543FCB for ; Fri, 19 Sep 2003 10:33:20 -0700 (PDT) (envelope-from rsharpe@richardsharpe.com) Received: from ns.aus.com (adsl-63-206-120-73.dsl.snfc21.pacbell.net [63.206.120.73])h8JHXIIO079420 for ; Fri, 19 Sep 2003 13:33:19 -0400 Received: from ns.aus.com (durable [127.0.0.1]) by ns.aus.com (8.12.8/8.12.8) with ESMTP id h8JHcUMI004970 for ; Fri, 19 Sep 2003 10:38:31 -0700 Received: from localhost (rsharpe@localhost) by ns.aus.com (8.12.8/8.12.8/Submit) with ESMTP id h8JHcTh0004966 for ; Fri, 19 Sep 2003 10:38:30 -0700 X-Authentication-Warning: ns.aus.com: rsharpe owned process doing -bs Date: Fri, 19 Sep 2003 10:38:29 -0700 (PDT) From: Richard Sharpe X-X-Sender: rsharpe@durable To: freebsd-hackers@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Throughput problems with NFS between Linux and FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 17:33:21 -0000 Hi, We recently encountered a problem with NFS throughput between a FreeBSD server (we are using 4.6.2, but the same code seems to be in 5.1 as well). When using Linux 2.4.19 or 2.4.21 as a client, although this might extend to other clients, and copying a large file, you will see the behavior shown in http://www.richardsharpe.com/ethereal-stuff.html#Time%20Sequence%20Graphs This happens because Linux hangs onto the ack for the last segment of a 32kB+header send for a while. The FreeBSD NFS server will not put anymore data in the socket because of an soreserve with a size of 32kB+header, so it waits for about 39mS until Linux finally sends the ack for the last segment. (Unless there is data, like another command, going the other way, that is). Throughput is about 3MB/s on GigE. The problem seems to be the following code if (so->so_type == SOCK_STREAM) siz = NFS_MAXPACKET + sizeof (u_long); else siz = NFS_MAXPACKET; error = soreserve(so, siz, siz); in src/sys/nfs/nfs_syscalls.c. We added a sysctl to allow finer control over what is passed to soreserve. With the fix in, it goes up to around wire speed when lots of data is in the cache. This was found by Chandu Gadhiraju with help from others. Regards ----- Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, sharpe[at]ethereal.com, http://www.richardsharpe.com