From owner-freebsd-net@FreeBSD.ORG Thu Jan 13 23:34:06 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B63D616A4CE; Thu, 13 Jan 2005 23:34:06 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id C28EE43D46; Thu, 13 Jan 2005 23:34:05 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 8CA887A403; Thu, 13 Jan 2005 15:34:04 -0800 (PST) Message-ID: <41E7056B.4010805@elischer.org> Date: Thu, 13 Jan 2005 15:34:03 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Andre Oppermann References: <41E5C9D8.4090209@elischer.org> <20050113055111.GA11141@odin.ac.hmc.edu> <41E6C564.50005@elischer.org> <41E6EADC.2AFAEA47@freebsd.org> In-Reply-To: <41E6EADC.2AFAEA47@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: net@freebsd.org Subject: Re: TCP out-of-order packets. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2005 23:34:06 -0000 Andre Oppermann wrote: >Julian Elischer wrote: > > >>Brooks Davis wrote: >> >> >> >>>On Wed, Jan 12, 2005 at 05:07:36PM -0800, Julian Elischer wrote: >>> >>> >>> >>> >>>>I have a link which is provided by someone else that is 7 x E1s aggregated. >>>>At leat it looks that way to me when I get to see it. however I have >>>>only been able to get >>>>60kB.sec across this, despite having a tcp window size of 131072 bytes.. >>>>After investigation it appears that the link is massively re-orderring >>>>packets. >>>>groups of upto 10 packets may appear in random order. (Maybe more, bu tI >>>>have seen 10) >>>> >>>>in fact packets are rarely IN order. >>>> >>>>This plays havoc with the tcp sessions. >>>> >>>>I was thinking of writing a hacked up version of NATD that >>>>instead of doing NAT, just did a pre-sort on packets from each session, >>>>so that the receiver would >>>>see a stream of IN-order packets, with occasional delays. >>>> >>>>firstly, does anyone have any tools to do this already (why build when >>>>you can borrow) >>>>and secondly, does anyone have any experience with this sort of problem? >>>> >>>>I have no control over or access to the link.. all I have is a promise >>>>that they will deliver >>>>14Mb/Sec. with approc 300mSec. RTT to me but there is no promise about >>>>packet order. >>>> >>>>I just get a 100Mb ethernet cable. >>>> >>>> >>>> >>>> >>>> >>>>I wonder if there's a way to turn off the sender backoff? >>>> >>>> > >Not directly. What you actually want is to delay the generating of >ACK's for a certain amount of time (some milli-seconds) to aggregate >the out-of-order packets into one ACK and to avoid the backoff from >the other side. > > yes, here's a trace from the receiver's end showing the order of received packets.. RTT on this link is about 300mSec. My original answer would be to hack NATD to re-order the packets. and use DIVERT to it. DeltaT SEQ-SRTR:SEQ_END packet# ------------------------------------------ 346853 2856:2920(64) 1 370821 2920:4368(1448) 2 004410 8712:10160(1448) 6 007848 12608:14056(1448) 9 007057 18400:19848(1448) 13 004027 11160:12608(1448) 8 005553 4368:5816(1448) 3 002390 16952:18400(1448) 12 005745 7264:8712(1448) 5 001204 5816:7264(1448) 4 008326 14056:15504(1448) 10 002555 10160:11160(1000) 7 004091 19848:21296(1448) 14 004287 15504:16952(1448) 11 358289 22744:24192(1448) 16 001891 21296:22744(1448) 15 048289 4368:5816(1448) DUP of 3 025538 5816:7264(1448) DUP of 4 018761 10160:11608(1448) DUP of 7 062939 25640:27088(1448) 18 005493 15504:16952(1448) DUP of 11 007903 24192:25640(1448) 17 002207 27088:28536(1448) 19 028675 21296:22744(1448) DUP of 15 176400 28536:29984(1448) 20 020889 29984:31432(1448) 21 092304 24192:25640(1448) DUP of 17 038396 31432:32880(1448) 22 015146 32880:34328(1448) 23 010362 27088:28536(1448) DUP of 19 027400 35776:37224(1448) 25 008269 28536:29984(1448) DUP of 20 012537 34328:35776(1448) 24 294035 37224:38672(1448) 26 078059 34328:35776(1448) DUP of 24 267701 40120:41568(1448) 28 017393 38672:40120(1448) 27 019769 41568:43016(1448) 29 339347 44464:45912(1448) 31 003887 43016:44464(1448) 30 123755 45912:47360(1448) 32 228883 47360:48808(1448) 33 013437 48808:50256(1448) 34 219579 50256:51704(1448) 35 144501 51704:53152(1448) 36 001200 53152:54600(1448) 37 029175 54600:56048(1448) 38 183158 56048:57496(1448) 39 157127 58944:60392(1448) 40 By here the sender has backed right off. > > >