From owner-freebsd-net@FreeBSD.ORG Thu Jan 13 19:31:38 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 65F7916A4CE for ; Thu, 13 Jan 2005 19:31:38 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3111C43D39 for ; Thu, 13 Jan 2005 19:31:36 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 217FA7A403; Thu, 13 Jan 2005 11:31:36 -0800 (PST) Message-ID: <41E6CC97.90603@elischer.org> Date: Thu, 13 Jan 2005 11:31:35 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Brooks Davis References: <41E5C9D8.4090209@elischer.org> <20050113055111.GA11141@odin.ac.hmc.edu> <41E6C564.50005@elischer.org> <20050113190953.GC28303@odin.ac.hmc.edu> In-Reply-To: <20050113190953.GC28303@odin.ac.hmc.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: net@freebsd.org Subject: Re: TCP out-of-order packets. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2005 19:31:38 -0000 Brooks Davis wrote: >On Thu, Jan 13, 2005 at 11:00:52AM -0800, Julian Elischer wrote: > > >>Brooks Davis wrote: >> >> >> >>>On Wed, Jan 12, 2005 at 05:07:36PM -0800, Julian Elischer wrote: >>> >>> >>> >>> >>>>I have a link which is provided by someone else that is 7 x E1s >>>>aggregated. >>>>At leat it looks that way to me when I get to see it. however I have >>>>only been able to get >>>>60kB.sec across this, despite having a tcp window size of 131072 bytes.. >>>>After investigation it appears that the link is massively re-orderring >>>>packets. >>>>groups of upto 10 packets may appear in random order. (Maybe more, bu tI >>>>have seen 10) >>>> >>>>in fact packets are rarely IN order. >>>> >>>>This plays havoc with the tcp sessions. >>>> >>>>I was thinking of writing a hacked up version of NATD that >>>>instead of doing NAT, just did a pre-sort on packets from each session, >>>>so that the receiver would >>>>see a stream of IN-order packets, with occasional delays. >>>> >>>>firstly, does anyone have any tools to do this already (why build when >>>>you can borrow) >>>>and secondly, does anyone have any experience with this sort of problem? >>>> >>>>I have no control over or access to the link.. all I have is a promise >>>>that they will deliver >>>>14Mb/Sec. with approc 300mSec. RTT to me but there is no promise about >>>>packet order. >>>> >>>>I just get a 100Mb ethernet cable. >>>> >>>> >>>> >>>> >>>Have you tried Andre's TCP reassembly rewrite? He says he saw >>>significant improvements in the face of major reordering. >>> >>> >>> >>> >>I don't think it's a problem with reassembly overhead, but rather a >>symptom of sender >>backoff when confronted with multiple duplicate acks due to the receiver >>getting the packets out of order. >> >>I wonder if there's a way to turn off the sender backoff? >> >> >> >>>http://www.mail-archive.com/freebsd-net@freebsd.org/msg14064.html >>> >>> >>> >>These machines are production machines on a custommer site.. running 4.8 >>it would be significant work to put the rewrite in (to 4.8) and a lot >>of red tape to >>reboot one to the new kernel.. :-/ >> >> > >Hmm, what about using a bridge with pf's TCP normalizer? You'd probably >have to raise your socket buffer sizes to handle the added latency (not >sure how much that is), but that might be an option (assuming adding >another machine wasn't harder then installing a new kernel). That code >is at least already written so you could also potentially crib from it >for the hacked up natd idea. > I have remote access to a handful of machines on a switch. I do not have access ot the switch or the wiring or the link.. for added points the machines are in India. > >-- Brooks > > >