From owner-freebsd-net@FreeBSD.ORG Fri Dec 10 23:05:36 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2336716A4CE; Fri, 10 Dec 2004 23:05:36 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5713A43D31; Fri, 10 Dec 2004 23:05:33 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.1/8.13.1) with ESMTP id iBAN5W0Y018539 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 10 Dec 2004 18:05:32 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id iBAN5RZF047494; Fri, 10 Dec 2004 18:05:27 -0500 (EST) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16826.11191.630110.592646@grasshopper.cs.duke.edu> Date: Fri, 10 Dec 2004 18:05:27 -0500 (EST) To: Andre Oppermann In-Reply-To: <41BA0088.9000107@freebsd.org> References: <41BA0088.9000107@freebsd.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-net@freebsd.org Subject: Re: Rewritten TCP reassembly X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Dec 2004 23:05:36 -0000 Andre Oppermann writes: > I've totally rewritten the TCP reassembly function to be a lot more > efficient. In tests with normal bw*delay products and packet loss > plus severe reordering I've measured an improvment of at least 30% in > performance. For high and very high bw*delay product links the > performance improvement is most likely much higher. > I ran netperf with 20 times for each of 3 socket buffer sizes (128KB, 256KB, and 1MB) before and after patching. All tests were run with net.isr.enable=1, and machdep.cpu_idle_hlt=0. CPU was pretty much maxed thoughout. (SMP kernel, 1 HTT p4). I'm not sure how much weight you should give these result. I still haven't gotten around to fixing kttcp, so the memory copy overhead is pretty high. For xmit, b/w goes from 2.90Gb/sec to 3.95Gb/sec when sendfile is used. (linux receiver on same hw). I've also not gotten around to making any of Robert's suggested driver optimizations. I noticed that at least systat was not printing the number of tcp out-of-order packets after applying the patch. I forgot to check netstat itself. Drew x before.131072 + after.131072 +--------------------------------------------------------------------------+ | ++ x + x x| |+ *+ + + ++ +xx++ x +x* ++ + xx+x x x + x x xxxx| | |_____________AM|___________|_______AM__________________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 20 3336.63 3516.95 3449.87 3447.838 52.708815 + 20 3323 3466.34 3391.97 3389.686 37.154513 Difference at 95.0% confidence -58.152 +/- 29.1859 -1.68662% +/- 0.846499% (Student's t, pooled s = 45.5998) x before.262144 + after.262144 +--------------------------------------------------------------------------+ | + | | + x | |+ ++ + ++ x x x | |+ + ++++++++ + + x x x x x xx xxx xxxx x x| | |______A______| |_______AM_______| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 20 3197.77 3421.67 3316.64 3314.8555 59.787371 + 20 2889.64 3122.17 2965.01 2964.9865 50.076867 Difference at 95.0% confidence -349.869 +/- 35.2961 -10.5546% +/- 1.06479% (Student's t, pooled s = 55.1463) x before.1048576 + after.1048576 +--------------------------------------------------------------------------+ | x + | |xx x + x xx + x+x++x * x* x+*x+x+* x*+ + + + ++| | |_________________|A____M__________A__M____________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 20 2658.05 2732.45 2706.91 2700.1955 23.318833 + 20 2676.13 2748.97 2724.07 2719.9665 20.531036 Difference at 95.0% confidence 19.771 +/- 14.0613 0.732206% +/- 0.52075% (Student's t, pooled s = 21.9692)