From owner-freebsd-net@FreeBSD.ORG Mon Dec 13 21:34:21 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE0A916A4CE for ; Mon, 13 Dec 2004 21:34:21 +0000 (GMT) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id B66B443D31 for ; Mon, 13 Dec 2004 21:34:20 +0000 (GMT) (envelope-from andre@freebsd.org) Received: (qmail 21567 invoked from network); 13 Dec 2004 21:23:19 -0000 Received: from unknown (HELO freebsd.org) ([62.48.0.53]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 13 Dec 2004 21:23:19 -0000 Message-ID: <41BE0ADA.9A1EE748@freebsd.org> Date: Mon, 13 Dec 2004 22:34:18 +0100 From: Andre Oppermann X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Andrew Gallatin References: <41BA0088.9000107@freebsd.org> <16830.1004.797220.47672@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-net@freebsd.org Subject: Re: Rewritten TCP reassembly X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Dec 2004 21:34:21 -0000 Andrew Gallatin wrote: > > Andre Oppermann writes: > > > > I've got some excellent review feedback from Mike Spengler and he found > > a off-by-one queue limit tracking error. > > > > http://www.nrg4u.com/freebsd/tcp_reass-20041213.patch > > > > Here are the same tests running your new patch in comparison to a > stock 6.x kernel from Friday. It looks like everything is faster. Excellent! > I've never seen 3.7Gb/sec on FreeBSD before. Nice work! Thanks! > The "before" and "tcp_reass-20041213" kernels differ only in the > contents of the above patch. I ran netperf with 20 times for each of > 4 socket buffer sizes (64KB, 128KB, 256KB, and 1MB) before and after > patching. All tests were run with net.isr.enable=1, and > machdep.cpu_idle_hlt=0. CPU was pretty much maxed thoughout. (SMP > kernel, 1 HTT p4). All the numbers in Mb/sec. The sender was running > linux-2.6.6 (also SMP on an identical HTT P4). > > The out-of-order accounting bug seems to be gone. Perfect. I want to get this patch into the tree next. I have already the next round in the works which is optimized even more by merging consecutive mbuf chains together (at the moment I have packet segment chains which have a direct pointer to the mbuf at the end of the chain) and which get passed in one go to soappend_stream. This removes the "present" loop and simplifies the general code a bit more again. With this and two other optimizations I have in mind you should be able to get very close to the theoretical maximum bandwidth of your current 4Gig Myrinet cards. There are a couple of other TCP tweaks that would help your special case some more now though. -- Andre