From owner-freebsd-net@FreeBSD.ORG Tue Mar 8 07:04:27 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 407F816A4CE for ; Tue, 8 Mar 2005 07:04:27 +0000 (GMT) Received: from angryfist.fasttrackmonkey.com (angryfist.fasttrackmonkey.com [216.223.196.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7CD8943D4C for ; Tue, 8 Mar 2005 07:04:24 +0000 (GMT) (envelope-from spork@fasttrackmonkey.com) Received: (qmail 76321 invoked by uid 2003); 8 Mar 2005 06:58:49 -0000 Received: from spork@fasttrackmonkey.com by angryfist.fasttrackmonkey.com by uid 1001 with qmail-scanner-1.20 (clamscan: 0.65. Clear:RC:1(216.220.116.154):. Processed in 0.049932 secs); 08 Mar 2005 06:58:49 -0000 Received: from unknown (HELO localhost) (216.220.116.154) by 0 with (DHE-RSA-AES256-SHA encrypted) SMTP; 8 Mar 2005 06:58:49 -0000 Date: Tue, 8 Mar 2005 02:04:20 -0500 (EST) From: Charles Sprickman X-X-Sender: spork@oof.local To: Daniel Hartmeier In-Reply-To: <20050307204825.GY26999@insomnia.benzedrine.cx> Message-ID: References: <20050305024850.GA96307@wjv.com> <20050307090802.GR26999@insomnia.benzedrine.cx> <20050307204825.GY26999@insomnia.benzedrine.cx> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: freebsd-net@freebsd.org Subject: Re: FreeBSD 4.x and OS-X tcp performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2005 07:04:27 -0000 Hello all, As promised, I have 4 tcpdump traces (saved with the "-w" option per request of another poster). Since these are a little too big to be broadcast to everyone on the list, I'm posting them here: http://home.manymonkeys.com/tcpdump/ For all tests I used os-x's command line ftp program and uploaded a 1.7MB file to each system. tcpdump.osx-fbsd - tcpdump ran on os-x box while transferring to FBSD tcpdump.fbsd-osx - tcpdump ran on FBSD box while transferring from os-x tcpdump.osx-obsd - tcpdump ran on os-x box while transferring to OBSD tcpdump.obsd-osx - tcpdump ran on OBSD box while transferring from os-x So there's a view of a single file transfer on each run of the test. Observing this showed that the os-x to fbsd transfer went at about 200KB/s and the os-x to obsd transfer went at about 2.6MB/s. Again I'd like to note that the OpenBSD box is at 3.3, which may prove important since it's quite dated. If there's anything else I can provide, let me know. Thanks, Charles On Mon, 7 Mar 2005, Daniel Hartmeier wrote: > On Mon, Mar 07, 2005 at 02:04:01PM -0500, Charles Sprickman wrote: > >> Very interesting, thank you for that read of the tcpdump output. If you >> have the time, could you post back a few lines of the tcpdump with >> comments so that I might learn a little about what's going on? I don't >> have the best understanding of the intricacies of tcp... > > Data is flowing only in one direction (from client to server), so we > see the client sending payload and the server acknowledging it. > > As long as no packets are lost and the server keeps acknowledging, the > client could send one continuous stream of maximum sized payload packets. > However, if the server stops acknowledging, the client must stop sending > more data once it has filled up the window advertised by the server and > wait for the server to either acknowledge further data or increase its > window. > > Nothing exciting happens until timestamp 16:41:29.008909 in your log. > > 16:41:29.008672 client 321457:322905(1448) > 16:41:29.008751 server ack 322905 win 7240 > 16:41:29.008909 client 324353:325801(1448) > > Since the log represents the server's view, the lack of segment > 324353:322905 in the log means that this packet from the client was lost > in transit. Now the server keeps acknowledging 322905, and we expect the > client to notice and retransmit the lost segment. > > 16:41:29.009064 server ack 322905 win 7400 > 16:41:29.009146 client 325801:327249(1448) > 16:41:29.009233 server ack 322905 win 8424 > 16:41:29.009409 client 328697:330145(1448) > 16:41:29.009521 server ack 322905 win 9448 > > The client hasn't noticed yet and is still sending further segments past > the gap. The server is advertising increasing window sizes, probably > because it's still draining the data up to the gap. > > Notice that there's a new gap created as 327249:328697(1448) was lost. > > 16:41:29.011250 client 335937:337385(1448) > 16:41:29.011331 server ack 322905 win 14568 > > Now the client has filled up the window. If it doesn't retransmit now, > it has to stall. It can't send anymore higher segments because the > window is full. > > 16:41:29.011758 client 322905:324353(1448) > > Finally, the retransmit. > > 16:41:29.011919 server ack 327249 win 10224 > > Now the server acknowledges up the the second gap. > > 16:41:29.021620 server ack 327249 win 13296 > ... > 16:41:29.024911 server ack 327249 win 56304 > > And keeps acknowledging the same, while the window grows again. But > nothing from the client. > > 16:41:29.146218 192.168.0.40.58347 > home.manymonkeys.com.ftp: . ack 376 > win 65535 (DF) [tos 0x10] > > Odd, an acknowledgment on the FTP control connection just at this time. > I don't know what that means, but it's not a coincidence, I'd bet. > > 16:41:30.462822 client 327249:328697(1448) > > Now the client finally retransmitted the segment from the second gap. > But look at the timestamp, about 1.4 seconds have passed, which is much > too long. > > After that, things go back to normal. > > The same thing repeats nine more times throughout the connection. You > can spot the places by comparing subsequent timestamps. You'll find the > spots where there are >1s delays in between two packets. > >> The tcpdump was run on the server (FBSD). Later today I will gladly do >> this again with a dump from each side. > > Before we blame the client, let's look at the dump from the client. > Maybe the client is trying to retransmit earlier, but those packets get > lost. It's just odd that this would happen for more than a full second. > > Daniel >