From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 3 08:15:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42ED6106569E; Fri, 3 Oct 2008 08:15:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 150E08FC08; Fri, 3 Oct 2008 08:15:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTP id 930A546B09; Fri, 3 Oct 2008 04:15:01 -0400 (EDT) Date: Fri, 3 Oct 2008 09:15:01 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Danny Braniss In-Reply-To: Message-ID: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , freebsd-stable@freebsd.org, Claus Guttesen Subject: Re: bad NFS/UDP performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2008 08:15:02 -0000 On Fri, 3 Oct 2008, Danny Braniss wrote: >>> it more difficult than I expected. >>> for one, the kernel date was missleading, the actual source update is the key, so >>> the window of changes is now 28/July to 19/August. I have the diffs, but nothing >>> yet seems relevant. >>> >>> on the other hand, I tried NFS/TCP, and there things seem ok, ie the >>> 'good' and the 'bad' give the same throughput, which seem to point to UDP >>> changes ... >> >> Can you post the network-numbers? > so I ran some more test, these are for writes IO: OK, so it looks like this was almost certainly the rwlock change. What happens if you pretty much universally substitute the following in udp_usrreq.c: Currently Change to --------- --------- INP_RLOCK INP_WLOCK INP_RUNLOCK INP_WUNLOCK INP_RLOCK_ASSERT INP_WLOCK_ASSERT Robert N M Watson Computer Laboratory University of Cambridge > > server is a NetApp: > > kernel from 18/08/08 00:00:0 : > /----- UDP ----//---- TCP -------/ > 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s > 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s > 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s > 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s > 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s > 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s > 64*512 602 0.20s 79.98MB 0.20s 78.44MB/s > 128*512 301 0.18s 86.51MB 0.22s 71.53MB/s > 256*512 150 0.19s 82.83MB 0.20s 78.86MB/s > 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s > 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s > 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s > 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s > 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s > 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s > 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s > > kernel from 19/08/08 00:00:00: > 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s > 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s > 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s > 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s > 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s > 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s > 64*512 602 0.43s 37.23MB 0.25s 63.57MB/s > 128*512 301 0.51s 31.39MB 0.26s 62.70MB/s > 256*512 150 0.47s 34.02MB 0.23s 69.06MB/s > 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s > 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s > 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s > 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s > 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s > 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s > 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s > > > > >